Microservices Summit
this is the 2nd microservices practitioners summit - there wasn't much from people who did it at scale
so we wanted to bring in people with that practical experience to talk to you today
20% people have 0 microservices in production - the rest are already running microservices
about 60% of people are interested in resiliency,
We start teh day with technology with @mattklein123 and @varungyan
the hashtag for today is #msvsummit - look for that
I'm Matt Klein, a software engineer at Lyft - I'm going to say how we got to a microservice mesh architecture
3-5 years ago Lyft had no SoA - an PHP/Apache monolith with MongoDB as the backing store with 1 load balancer
PHP/Apache's process per connection model doesn't talk to the load balancer, so we had problems there too
2 years ago we had an external LB, a php monolith wiht haproxy to call internal ELBs + a bunch of python services
we had problems with logging and tracing and understanding which layer something died in
in SoA now the industry had tons of languages and frameworks - 3-5 different languages in one deployment
also, there are per-language libraries for making service calls - Php uses curl, Java uses finagle etc
we have multiple protocols - http http2 gRPC, databases etc
we even have multiple infrastructures IaaS CaaS on premise and more
we een have multiple heterogenous load balancers, and very different observability of stats, tarcing and logging
we also end up with multiple implementations of retry, circuit breaking and rate limiting, often partial ones
if you're using all of this stuff in your 5 langs it can be impossible to know what is calling what
and Authn and Authz is often an afterthought, with no key rotation
People do not understand how all these components come together to build a reliable system
People are feeling a lot of hurt, especially around debugging
when I joined Lyft people were actually afraid of making service calls as they couldn't know what went wrong
you have limited visibility into different vendors logging and tracing models, so there is little trust
existing libraries often have partial implementations of best practices -
when we were building Envoy, people would ask why they needed this for retry
retry done wrong is the best way to bring down a system
if you do have a good answer, you are using a library that locks you into a technology stack
if you're invested in JVM and you want to use Go services, you need to port the big library over
if you have a big standard library, upgrading it to a new version can be a huge pain point
Robust observability and easy debugging are the most important thing - without that devs don't trust the system
we have not given people good tools in SoA to do this kind of debugging - so productivity goes down
when people don't trust the network of service calls, they rebuild monoliths with fixed libraries again
Envoy wants the network to be transparent to applications, which is really hard
Envoy is not a library, it is more like nginx or haproxy - it is its own process that is next to each application
the application talks to Envoy locally, Envoy does the work and moves stuff back to the applicaction
Envoy is in C++ and is a byte oriented proxy - it can be used for things other than HTTP - stunnel, redis, mongo
as well as that L3/L4 filter stack we have an L7 http filter architecture that lets you do header work too
Envoy was built to be http2 first, but with an http 1.1 bridge - we can proxy gRPC which is http2 based
we have servoce discovery and active/passive health checking
and advance load balancing with timeouts, circuit breaking rate limiting and so on
we have best in class observability of tracing and stas,
we have enough features to replace nginx as an edge proxy as well as in service to service mode
the model we have is many service clusters with an envoy instance with each service, talking to each other
and also using Envoy to call out to External services and discovery.
your service is only aware of your local Envoy, so doesn't change whether in local, dev or production
Envoy sets up the environment so that dev, staging or production just works - can mix local and cloud abstactly
we have 2 kinds of edge proxies - one terminating TLS and connecting our internal services,
but we also use Envoy to proxy between different geographic datacenters
we have an edge proxy Envoy, which calls the Envoy on our legacy monolith and python and go services
services don't talk to anything without talking to Envoy - it proxies DynamoDB and MongoDB too
The way that most service discovery works is fully consistent systems like zookeeper and consul
but service discovery isn't fully consistent - it changes over time
if you have a fully consistent problem that can be eventually consistent, make it eventually consistent
because service discovery is an eventually consistent problem, we designed it that way
we have about 300 lines of python that checks each server once a minute into a dynamodb
we have active healtheck checks every 15 seconds, and passive restart on fail
we trust the active health check more than the discovery service as it is lossy
if the health check fails we don't route; if the health check fails and discovery shows absent we remove the node
we have not touched our discovery service in 6 months because it does converge
people who do use zookeeper and etcd fully consistent systems build eventually consistent discovery on top
we have multiple service discovery models, including zone aware load balancing - local first then remote
we can generate dynamic stats, and also do circuit breaking and rate limiting too
we plan to open source the rate limiting service next week
we support shadowing so you can fork traffic to a test server
we ahve built in retries, and inner (1 service) and outer (whole callchain) timeouts
you can have all these features, but without observability no-one will use them
by routing all traffic through Envoy, we can produce stats, but also sample entire request chains
because we have a stable requestID, we can trace and log across multiple systems and servers
you can have a dashboard that shows all connections between any 2 services
this lets you look at any 2 hops in the system and how they relate
for all of the data transited through envoy, you can see the flow of requests through services by default
our logging system kibana uses the stable request id to connect all the different components and what happened
a lot of people say 'performance only matters if you are google' - dev time is more important
but latency matters, and people don't think about tail latency - the p99+ problem
we have a lot of tools that make programmers more productive, but much harder to see where time is being spent
throughput may not be the most important thing, but being abel to reason about where time is spent really matters
if the service proxy itself has tail latencies that are hard to reason about, you lose the debugging benefits
you don't want a proxy that adds latency variance and makes your debugging harder
Lyft has >100 services, >10,000 hosts and >2M RPS - we proxy gRPC, mongodb and dynamodb too
we sniff the mongodb and dynamodb traffic to generate stats on performance and latency
we are adding redis soon to reduce outliers
we are spending more time on outlier detection and rejection
we are working to standardize load balancing and rate limiting across services
Envoy has only been open source for about 4 months, but we have a lot of interest already
we want to build a larger community around Envoy
you can get the code at lyft.github.io/envoy
when you were getting lyft to switch over to Envoy, what was hard?
we started incrementally - Envoy as front proxy first. then we added envoy to the monolith, then on mongodb
we are now fully deployed but it took a year to get concurrent development
do you reserve a core for envoy?
you can do that, but it can make things work. Envoy is non-blocking and parallel. so run 1 thread per core
are data piplelines eg spark clusters integrated with Envoy?
we do use them fro LB but we don't use it directly for Spark
can you add filters?
we don't have any public docs on filters yet, but multiple companies have written them from the code
a disadvantage is bringing the work into the envoy team - how do you get it out again?
that hasn't been a problem so far - "if the word 'car' appears in Envoy we have done it wrong"
the filtering model is extensible enough that we haven't needed to block on Envoy
a lot of systems burn network bandwidth on health checks - do you watch response and health checks separetely?
active and passive health checks are configurable so you can decide which to use.
there is perception that active health checking is wasteful, but with plaintext kept-alive http1 it is very low
we run health checks every 15-30s and it is noise in our traffic graph
if it does have scale issues we are working on subsetting these so the traffic does't transit so much
there is no reason that the service discovery system couldn't do health checks too
I like to deploy microservices in docker containers, would that work for Envoy?
we support hot restart in Envoy so it can new code deploy without losing connections-that works fine in containers
we are livestreaming at microservices.com/livestream - follow along there
I'm Varun from Google - I'm here to talk about microservices at Google, but based on our gRPC experience
Stubby is an internal framework at Google for all service to service calls
we want to bring what we learned from Stubby into the newer open source gRPC
people want Agility and Resilience, and that is why we use microservices,
but we also care about developer productivity - as @mattklein123 said observabilty is key to trust
even a 400ms delay can have a measurable impact on search quality and usability
Stubby was an RPC - Remote Procedure Call - framework written when google started, used for all google services
Stubby is a large framework, and parts of it are being made more open
Google's scale is about 10^10 RPCs per second in our microservices
every Googler defined datatypes and service contarcts and magic around load balancing, monitoring and scaling
making google magic available externally - Borg became Kubernetes; Stubby became gRPC
HTTP1.x/JSON doesn't cut it at Google scale - stateless, text, loose contracts, TCP per request, nouns-based
a framework with tighter contracts, more efficient on the wire and with language bindings help a lot
when APIs are evolving at different rates, in classic REST this needs a lot of work in polyglot environments
from a pure compute perspective, having text on the wire isn;t most efficient
we needed to establish a lingua franca for strongly typed data - Protocol buffers released in 2003
declare your data in a common description format, and generate code for any language with fwd/backward compatibility
at Google you are either doing Proto to Proto, or Proto to UI - that's it
protobufs incrementally number fields in order of creation, so you can evolve dat structures over time
the other big protobuf advantage is carrying binary on the wire, giving a 2-3x improvement over JSON
designing for fault tolerance and control is key - sync vs async; deadlines and cancellations; flow control, metadats
different languages default to doing these in different ways, so moving to the core helps
in GRPC we have deadlines, not timeouts - each service will add on time taken and abort if it exceeds deadline
deadlines are expected, but we also need unpredictable cancellations too, when result not needed
with services you need cascaded cancellations that clears all dependent calls too
flow control is not a common occurrence, but matching a fast sender to a slow receiver or vice versa does matter
with gRPC when there are too many requests or responses there is a signal sent to slow down
you can set up service configuration policies that specifies deadlines, LB policy and payload size
the SREs like having this config separately so they can control when things are surging
you don't just need an RPC contract, but also to send metadata for AuthN and trace contexts etc
this metadata helps keep the control flow out of the specific APIs
you want observability and stats - you need a common nomenclature to make sense of large call graphs
you can go to any service endpoint and see a browser dashboard in real time on how much traffic is flwoing
Tracing request through services is key - you can add arbitrary metadata such as client type and track it all
you often have 1 query out f 10,000 that is slow - you want to trace it through the whole call chain
you also want to look at aggregate info to see where the hotspots are
Load Balancing matters, and you need to communicate between the front end, back end and the load balncers
gRPC-lb is moving to a simpler model where the client has a round-robin list of endpoints and the backend models them
gRP is at 1.0 now, and has more platforms and languages now it is in open source
gRPC has service definitions and client side libraries that bridge multiple languages and platforms
http2 is the basis which give streaming and much better startup performance
coming soon: reflection; health checking; automated mock testing
it's at grpc.io and github.com/grpc
how did you switch from Stubby?
that si still happening at google, as a lot of gRPC benefits are already in Stubby
we have to show RoI to service owners inside Google. If you don;t have Stubby, thev alue is clearer
how do you do the proto sharing and distribution? one concept of 'user' or every service has one?
do you share contracts between services?
every service defines its own, apart from things like tracing and logging
do you generate client and service code from protobufs? does it limit flexibility?
yes we generate both; all we generate are stubs i that language - whatever you define is what you get
we try to make our APIs as close as possible to the language you are using, so we have fututes in node etc
I'm Christian Posta from Redhat talking managing data inside microservices slides at bit.ly/ceposta-hardest-part
I commit and contribute to apache projects like Camel, ActiveMQ and Kafka
I used to work at a large webscale microservices unicorn company, now I bring that to enterprise
when developers approach Microservices or what was called SoA, they ned to think about more than infrastructure
Adrian Cockcroft warns that you need to copy the Netflix process, not just the results of it
when Enterprise IT approaches this microservices world, there is a mismatch of culture
microservices is about focusing on speed - in terms of being abel to make changes to the system in production
IT in Enterprise has always been seen as a cost center, or as a way of automating paper processes
how do you change a system not designed with this kind of iteration in mind to go fast?
we need to think about managing dependencies between teams as well as services
the difficulty of data is that it is already a model of the world - it hasn't got the human context
even something as simple as a book ends up in multiple models - editions, physical copies, listings by author
each service looks at these things a little bit differently - Domain Driven design helps with this
domain driven design means breaking things into smaller understandable models & defining boundaries around them
enterprise models can end up more complex than purely virtual companies, as the models map to processes
if you write a lot of denormalised data into your databases, you need to plan for the queries you're running
Keep the ACID and relational conveniences as long as you can, but be aware of what they cost
with microservices we're saying "OK, database workhorse, we've go it form here"
saying "a microservice has its own database" sounds very worrying to an enterprise data modeller
microservices means taking concepts of time, delay and failure seriously rather than ignoring them
when we need to distributed writes to maintain consistency, we end up building a transaction manager
it is easy to accidentally build an n+1 problem into your services, and you end up adding extra calls to fix
with CAP, you can't trade off P, so you have to pick C or A - but there are lots of consistency models
you don't need strict linear consistency if there is no causal relationship between entities
in real life there is very little strict consistency—think how paper processes propagate updates over time
a sequential consistency is often a good answer - a log or queue to process data in order
when we do this, we have made the replication and indexing that databases did for us an explicit process
Yelp has mySQL Streamer, LinkedIN has Databus, Zendesk has Maxwell - all do this queue to DB model
There's a project debezium.io that captures database changes and streams them in a queue to something like Kafka
I'm going to do a live demo of debezium.io
what were you going to demo?
I was going to start up kafka and a mysql database, and connect them with debezium
debezium captures the primary key of the table and uses this to send the before and after changes to the DB
so we can show a change to the database from mysql binlogs in a JSON form in a kafka queue
how do we make this work in an environment without arbitrary extentions, like Postgres?
we are working on adding Postgres log support - there was a PR for that recently
when you mentioned Domain driven design is CQRS in play?
CQRS is separating different read/write workloads into different data systems
if your reads are simpler you could use this to transform data into a denormalised system
we've had around 1000 people tuned in to the stream at https://www.microservices.com/livestream/ and we're starting again
Microservices are the Future, and always will be
xoom.com is a digital remittance company founded 2001, acquired bluekite in 2014, joined paypal 2016
remittance is sending money between countries - we go between 56 countries at the moment
as a finance company, we have very strict regulatory compliance, both in the US and the 55 other companies
we have 16 years of code and 16 years of data to migrate into our microservices
we have lots of code and lots of tables, and code that assumes all those tables are joinable
xoom was an all java shop, but bluekite made us polyglot - we have many languages and persistence techs now
Paypal acquiring us imposed new rules on us, but they also are used to a polyglot environment
we wanted to break up the monolith when we hit build time limits on our SQL based infrastructure
so a few years ago we started to decouple the teams to reduce the build times
we wanted to understand which parts of our stack were the bottlenecks, and scale them appropriately
moving to microservices we had to change a lot of programming paradigms and idioms
we needed service discovery and monitoring to understand performance
we had snowflake code all over the place in our load balancers and deploy path
we needed to switch to a unified build and deployment pipeline
and we needed to pick apart our databases and define the data ownership and contracts
microservices can be a distraction to our engineers - things like circuit breakers and throttles are hard
API designs need thought - the N+1 problem needs thinking about, and RPC vs REST adds complexity too
with API designs, response code granularity can be a huge issue too - do you pick just some http response codes?
Contracts are a key point if you are polyglot coders - you need strong contracts for packaging and metadata
we use docker containers for microservices; each service has metadata on the containers and runtime introspection
we include things like pager rota in the metadata so you can introspect them to know who to call
we can monitor and manage the instances uniformly with this model
Having polyglot code can slow you down; if you can stay in a single language and db, do it
our service discovery is similar to what Envoy is doing,
we have a custom layer 7 load balancer - zoom.api resolves to the local network
so I can hit auth.2.zoom.api and get all the instances with reputation based routing
we have a zookeeper backend , but we have an eventually consistent layer on top of it
we have a service portal that shows the services and health checks adn routing for each one
if you integrate k8s, you need to think about external vs internal service discovery and ip routing
for monitoring we initially set up a graphite system and threw lots of data at it, and crashed it with load
we were trying to instrument time taken on every call, and this was enough to overwhelm graphite
we chose to use http and json for our internal calls between apis
a call that involves a post and a write as opoosed to a read, is much more complex to monitor
we built a time series for every endpoint and call, and that also created a lot of traffic
we used the dropwizard monitoring library which gave use gauges and histograms as well as counters
we were very worried about performance when we started on this journey - we were worried about extra net traffic
we spent a lot of time instrumenting our code before we made any changes, and I recommend that
we improved the throughput of our service dramatically, primarily because of the shift to accurate monitoring
this helped us reduce contention over shared resources, despite making more RPCs overall
the latency distribution is wider now - we adjusted latency sensitive APIs to be deployed nearby
infrastructure as code matters - TDD isn't just fro code, write tests for deployment too
don't treat deployment code and networking configurations as special - they all need tests too
by standardising app packaging, we can have contracts for deployment too
we use git-flow for new features, with a container per branch using docker-flow, automated+ self service deployments
by standardising deployment pipeline, we can have a portal to enable PMs to deploy versions withtout ops
data ownership is hardest - we are eliminating cross-domain joins and adding apis to wrap them
we have about 100 different services now
the key is to measure everything, and be prepared to scale monitoring to cope
application packaging contracts and delivery pipelines are mandatory
staff a tooling team to build, test and deployment automation and bring in network ops
although our monolith is still partly there, the infrastructure and cultue has improved everything
what was the challenge you were least expecting?
the metric explosion caught us off guard - be prepared fro that
how do you do integration testing when you have a large number of services?
we have ~200 clusters we can spinup and down in both amazon and our data center
the other integration testing is to create mocks so you can run those. also can run containers locally and route to cloud
for us, anyone who writes code can't touch the production network and vice versa - this is not very DevOps
sounds like you built togetehr multiple solutions. how do you monitor end to end?
for our java applications we wrote a wrapper applications to give tracing for free
calico gives us the ability to have routable ip addresses per pod, which helps us with monitoring the whole system
we found that we had 2 services that constantly want to make joins, so they need to be one service not two
you mentioned that this was a cultural change, what was the impact?
our product managers have been very customer focused. The big change was getting them to think about SLA and contracts as well
were there challenges in batch jobs?
putting the batch jobs with the domain that owns them makes more sense
I'm Rafi Schloming from datawire - we founded it in 2014 to focus on microservcie from a distributed systems background
I participated in every version of AMQP and had built lots of distributed systems with them, so I thought it would be easy
I wanted to look back at my learning about microservices
wikipedia isn't helpful here - "there is no industry consensus" "Processese that communicate" "enforce modular naturally"
there are a lot of good essays about microservices, but also a lot of horror stories of going wrong
the 3 aspects I want to cover is the technology, the process and the people
we learned from experts, from bootstrapping ourselves and from people migrating to microservices from many origins
3 years ago it was very technically focused - a network of small services, hoping it would make better abstractions
we read every story of microservices, went to conferences, started the summit ourselves to share the ideas
the people picture: everyone has a developer happiness/tooling/platform team and a service team that build features
technically we saw a control plane for instrumenting the services , the services and a traffic layer
it's a lot of work to build a control plane, so we decided to provide that as a service for the teams
so we ingest interesting application events - start, stop, heartbeat. log these and register services; transform & present
we were building a classic data processing pipe line of ingest, write source of truth, transform and present
for version 1 we built discovery - highly available, low throughput and latency; low complexity and able to survive restart
we started with vert.x and hazelcast and websockets with smart clients
for version2 we added tracing - high throughput , a bit higher latency was OK
version 3 we added persistence for tracing by adding elastic search
this was the 1st hint of pain - we had to reroute data pathways and had coupled changes, and this gave a big scary cutover
v4: we added persistence for discovery, using postgres for persistence, which was another scary cutover -lets fix our tools
Deployment was hard. we had tried docker, but that was hard to bootstrap; kubernetes required google not amazon
we redesigned our deployment system to define the system in git to bootstrap from scratch
this meant we could use minikube locally with postgres and redis in docker images
and then spin this up to production running in amazon with our own kubernetes cluster
we built tooling to make this work across the different dev and deployment environment
did we just reinvent DevOps the hard way? we were thinking about operational factors, we built a service not a server
rather than a Service Oriented Architecture, we had a Service Oriented Development
Architecture has lots of upfront thinking and a slow feedback cycle. Development is more incremental
Development is frequent small changes with quick feedback and measureable impact at each step
so microservices are a developmental methodology for systems, rather than an architectural one
small frequent changes and rapid feedback and visibility are given for a codebase, but harder for a whole system
so microservices are a way to gather rapid feedback - not just tests but live measurement
instead of build - test -deploy we want build - test - assess impact - deploy
so measure throughput, latency, and availability measured as error rate
the experts model of canary testing, circuit breakers and so on are ways of making sense of a running system
Technical: small services; scaffolding for changes Process: service oriented development People: tools and services
working with people migrating gave us much more information
migration is about people. Picking a technical stack for the entire Org is hard; refactoring has lots of org friction
creating an autonomous team to tackle a problem in the form of a service is much easier
some organisations hit a sticking point, others didn't slow down
the way to think about microservices is in dividing up the work: build features (dev) Keep it running (ops)
you can't easily divide along these lines - new features make it unstable. devops stops misaligned incentives
microservices divides up the work - a big app made of smaller ones, that are easier to keep running, aligning incentives
if you think about microservices an an architecture you forget about the operational side of keeping them running
the easy way: start with principles of People and Process, and use that to select the technology
how would you boil this down to one statement?
start with the people and think how to divide up the work first, let that lead to the technical perspective
how much time did you spend on research and things that didn't make production?
it's hard to quantify that time spent - it ended up as a fragmented and incremental view
do you see Conways law affecting your team size?
yes, there is an impact there - trying to fit the information into the picture
that the shape of the team drives he shape of the technology is true, but physics pushes the other way
I'm Nic Benders, chief architect at New Relic, talking about Engineering and Autonomy in the Age of Microservices
I want to talk about that you can accomplish in an engineering org with microservices
New Relic started out with a data collection service and a data display service that started out micro and grew
we now have over 300 services in out production environment
Conway's law always is in play - our production environment reflects the communications & dependecies between teams
Conway's law is about how teams communicate, not the actual org chart. It's the edges, not the nodes that matter
Conway: Organisations are constrained to produce designs that are copies of the communication structures of the org
microservices is meant to define teams around each service - that is the core
componetisation via teams organised around business capabilities - products not projects so long term ownership
smart teams and dumb communication pipes - use a lightweight tool like a wiki or blog
durable full-ownership teams organised on business capabilities w authority to choose tasks & complete independently
reduce central control - emphasising information flow from the center and decision making at the edge
Eliminate dependencies between teams as each dependency is an opportunity to fail
having a re-org seems like a good idea, but it doesn't really work well if you just rename and change reports
what if we look at an org structure as an engineering goal? Optimize for agility -not utilisation of a team
if you optimize teams for efficient usage of the team, you make sure that they have a backlog to keep busy
what we need are short work queues and decision making at the edge
as chief architect, I know far less than about the domain than the engineer working on the problem does
at new relic, we're data nerds. We should use data to make our changes, not VPs in offsites
the most important thing in our org change is to break our dependencies betwen teams
we drew the nodes as teams and the edges as dependencies, and simplified universal ones
we proposed some much simpler dependency diagrams, with fewer, stronger teams with full ownership
in a full stack team, you are missing a business ownership component, so we added PMs and tech leads for internal
for the team to work it needs more T-shaped people. with depth in one area, and breadth across others
we abolished architecture reviews, and made each team stand alone and own its decisions
we decided to allow team self-selection, as people know their skills better than we do
we put out all the jobs needed by the department and the engineers pick the ones to do. This is harder than it looks
Managers really didn't like this. Managers tend to follow instructions anyway, so it worked.
Engineers didn't like it either. They didn't trust us -they thought there would be fewer jobs that they wanted
they also worried that they would pick the wrong thing, or that the teams would gel without managers
We almost backed down. But we had to get the teams to self correct. We had failed to empathize with their concerns
we had to communicate over and over that this wasn't a stealth layoff or a job fair, but we would take care of them
we were not shifting the burden of being responsible to the employees but making sure we still looked after them
we defined the teams & the skills they needed, not in terms of positions & got everyone in a room to find new teams
at this point we had at least made it clear that there were other teams that you could move to
about a third of the people there did switch teams - lots of new teams formed from scratch
working agreements per team were defined as "we work together best when…" for them to fill in
the insights team picked Continuous Deployment Weekly demos and Retros, and Mob Programming
Mob Programming is like pair programming, but with 6 people sitting round the computer with 1 typing - huge agility
this reorg really worked - we shipped far more this year than expected, because they worked faster on what mattered
Teams understood their technical remit, but not what the boundaries were - we were used to side projects
we wrote a rights and responsibilities document - teams write own Minimal Marketable Features, but must listen too
maybe you aren't going to try a 6-month re-org, but there are takeaways
you hired smart engineers - trust them. We didn't do this with MBAs and VPs but with the teams themselves
my presentation is up at http://nicbenders.com/presentations/microservices-2017/ and you can tweet me with comments
the main thing I was worried about tactically was making 300 people fit into 300 jobs and teams would not fit
there were a few people in critical roles that we had to keep in place, and we really owe them
what does a manager do in this kind of team?
regardless of org structure, managers need to look after their team and the teams' careers
we have spent more time encouraging the embedded PM to work with the team since
what happened giving the teams total control of technology?
I sometimes think "that's not the best tech really, it's just some hacker news thing" but if they can go faster…
we have some constraints - we are container based, but it you need, say, an Elixir agent you have to build that too
you mentioned 6 months - was that how long it took to settle down?
within the 1st month there were teams up and running, but the experience varied.
did the managers go and find new teams, or were they fixed?
in general managers were the core of the team, and engineeers moved to them, which may be why they were unhappy
how did this map to employee performance management?
we did reset the performance history, and had a lot of success, and a modest turnover too close to annual average
who owns the space between the teams? how do they call each others code?
communication is owned by the architecture team, and we have cross-functional groups for each language etc
we have a product council to say what the key products and boundaries are but no the detail which team does
we mapped every product and service, including external ones, before we moved everyone
we had a 2 week transition to make the pager rotation handovers and deploy work.
after this do the people feel they need another team change? how often do you redo this?
it was such a production that we would rather have a continuous improvement rather than an annual scrambling
we have a quarterly review per team, but we want to make it possible for internal transfers be low friction
I'm Susan Fowler here to talk about Microservice Standardization
I stared off thinking I would do particle physics forever, but there are no jobs in physics
I worked on the Atlas experiment at CERN, and then went to Uber to work on their 1000 microservices
there were some microservices that had lots of attention, but we were the SRE microservices consulting team
I also wrote a book called Production-Ready Microservices - there is a free summary version online
every microservice orgnaisation hits 6 challenges at scale
challenge 1: Organisation siloing and sprawl - microservices developers become like services and are very siloed
unless you standardise operational models and communication, they won't be abel to move teams
when you have too many microservices, you can't distribute ops easily, so the devs are fighting ops battles too
challenge 2: More ways to fail - more complex systems have more ways to fail. don't make each service a SPOF
challenge 3: competition for resources - a microservice ecosystem competes for hardware and eng resources
challenge 4: misconceptions about microservices - wild west; free reign; any lang; any db; silver bullet
myth: engineers can build a service that does one thing extraordinarily well, and do anything they want for it
a team will say "we heard cassandra is really great, we'll use that" the answer: "you are taking on the ops too"
microservices are a step in evolution, not a way out of problems of scale
challenge 5: technical sprawl and technical debt - favorite tools, custom scripts; custom infra; 1000 ways to do
people will move between teams and leave old services running - no-one wants to clean up the old stuff
Challneg 6: inherent lack of trust - complex dependency chains that you can't know are reliable
you don't know that your dependencies will work, or that your clients won't overload you; no org trust
no way of knowing that the microservices can be trusted with production traffic
microservices aren't always as isolated from each other as their teams are isolated
The way around these challenges is standardization. Microservices aren't isolated systems
you need to standardize hardware (servers, dbs) Communication (network, dbs, rpc) app platform (dev tools, logs)
the microservices and service specific configs should be above these 3 standard levels
a microservice has upstream clients; a database and message broker; and downstream dependencies too
the solution is to hold microservices to high architectural, organisational and operational standards
determining standards on a microservice by microservice basis doesn't establish cross-team trust, adds debt
global standardization company wide - must be global and general; hard to determine from scratch & apply to all
the way to approach standards is to think of a goal - the best one to start with is availability
availability is a bit high level, map that to stability, reliability, scalability, performance, monitoring, docs
microservices increase developer velocity, so more changes, deployments, instability - need deploy stability
if you have development, canary, staging, production deployment pipeline, it should be stable at the end
Scalability and Performance - need to scale with traffic and not compromise avaiabiliity
fault tolerance and catastrophe preparedness - they need to withstand internal and external failure modes
every failure mode you can think of - push it to that mode in production and see how it does fail in practice
Monitoring and Documentation standards mean knowing the state of the system, and very good logging to see bugs
Documentation removes technical debt- organisational level understanding of it matters
implementing standardisation needs buy-in from all org levels; know production readiness reqs and make culture
my free guide is http://www.oreilly.com/programming/free/microservices-in-production.csp and my book is at http://shop.oreilly.com/product/0636920053675.do
when you have a set of answers to the standards for the whole company, it makes it all easier
how long did it take to fix all the services?
It's still going on; we started with the most urgent services and helped them understand what was around tem
you mentioned proper documentation - how do you keep up with code changes?
that's a good question - keep docs to what is actually useful and relevant. Describe arch and endpoints
documentation isn't a post mortem for an outage, but an understanding of what works
is slicing up the layers a contradiction to devops?
the microservices team should be on call for their own services' outages but need to be both?