"The relentless pursuit of simplicity"
Book: Lean SW development (Poppendick)
All talks from Mary are on chapters from the book
The question: How do we deal with Complexity?
The complexity (and mess)of software is often invisible from the outside,
The answer: Divide and conquer
But how to divide - there are several options?
- Divide by Structure
(Divide by Hierarchy/order of execution)
Historical perspective on various ideas on how to structure. "Structured programming - Quality by construction", "Top-Down programming - Step-Wise Integration",
xUnit testing is an good answer to this approach, but n
- Divide by responsibility
(Divide by Matching the Domain)
- Divide by process
Example: The Waterfall process.
This has never worked well.
- The cost of debugging
- The cost of integration
- The cost of task switching
- The cost of delay
- The cost of complexity (to change)
- Divide by Value
(Design by objectives)
Let us not confuse the differences between Project Management and Systems Development
Project Management (Changes over time)
- Requirements
- Quality of testing
- Scope conrol
- Maturity levels
- Resources
- Timeboxes
Systems Development (Robust over time)
- Low dependency architecture
- Quality of construction
- Respect for complexity
- Technical disciplines
- Technical expertise
- Learning/feedback cycle
Pay attention to the robust part!
- Divide by responsibility and value = The agile idea.
Nice slide on the various "phases" and what to focus on in these (Goals/Roadmap, Iteration planning, Design and Implement, Delivery)
Emphasizes the benefit of the practice of writing tests first! Including xUnit tests and
Acceptance Tests, Stress tests, Presentation layer tests
Displayed a matrix with the different test types, proposed automation level and frequency + the two different perspectives of "Testing to specification" and "Testing to Failure"
Refactoring
- Why do we need to refactor?
Because we practice JustInTimeNOT JustInCase. This reduces complexity.
Refactor: Simplify the code base - eliminate smells
Need a policy to drive this - E.g. We will have clean and readable code
4 criteria for simplicity
- passes the test
- reveals all of the intentions
- no duplication
- fewest number of classes/methods
Respect for complexity
- Accept that change is not the enemy
- Recognize that complexity is the enemy
- Simplify before automating
- Write less code - essential simplicity
Example from Amazon.com
Did not consider the scale when they started..
Started transformation to services in 2000 - transform completed in 2008..
- each service is owned by a small team (about 8 people). No service is allowed to be bigger than possible to manage by this team.
The team is responsible for everything related to the service
(spec, dev, deploy, support, everything..)
SLA's and contracts towards other services
Very little standardization across
People talk and cooperate across teams, but is allowed and encouraged to make your own decisions
Technical discipline
Mistake proofing
- Design/code reviews
- Config/version management
- One click builds (private and public)
- CI
- Automated Unit Tests
- Automated Acceptance tests
- STOP if the tests dont pass
- system testing with eatch iteration
- automated release/install
- escaped defect analysis & feedback
Simplicity
- Low dependency architecture
- Development standards (coding, naming, logging, security, usability)
- Standard tools
- Refactoring
Technical expertise matter. Example from the pilot that landed in the Hudson river.
Must develop a practice to keep improving
- Realise what you want to improve
- Get exercises to improve them
- Focus on pushing the limits
- Practice regurlary and intensely
Create competencies leaders that focuses on evolving and nurturing peoples competences.
This is not the same as the Product Champion (example: Music Teacher VS The Maestro)
19 Jun 2009
Mike Cohn: User Stories
The technique of expressing requirements as user stories is one of the most broadly applicable techniques introduced by the agile processes. User stories are an effective approach on all time constrained projects and are a great way to begin introducing a bit of agility to your projects.
In this session, we will look at how to identify and write good user stories. The class will describe the six attributes that good stories should exhibit and present thirteen guidelines for writing better stories. We will explore how user role modeling can help when gathering a project's initial stories.
Because requirements touch all job functions on a development project, this tutorial will be equally suited for analysts, customers, testers, programmers, managers, or anyone involved in a software development project. By the end of this tutorial, you will leave knowing the six attributes of a good story, learn a good format for writing most user stories, learn practical techniques for gathering user stories, know how much work to do up–front and how much to do just–in–time.
Balance is critical (business and development)
- Technical vs business issues in focus
Resource allocation should be a shared problem (business and development)
Imperfect schedules
- We cannot perfectly predict a software schedule
- If we can't perfectly predict a schedule, we can't say what will be delivered
So what do we do?
- We make decisions based on information we have, but do it often
- We make decisions throughout the whole project
UserStories - Three C's of Ron Jeffries
- Card - The basics of the US
- Conversation - The details, captured in conversations with product owner
- Confirmation - Acceptance tests confirm the story was coded correctly. Defined by Product Owner in cooperation with team
Mike displayed UserStories examples
considers the "so that" clause as optional - use when needed to improve understanding, not when it is obvious
How to capture the detail?
Several options:
1. Write details as "conditions of satisfaction" - these are essentially tests, but we do not use this term (test) since it is misinterpreted by many..
2. Create a hierarchy with the details in sub-user stories
These techniques can and should be combined
Mental model for the product backlog as an iceberg with more detail/smaller items on the top (Sprints), bigger in the middle (Release), and large at the bottom (future releases)
Someone needs to regularly "grooming the product backlog". Estimated required effort for this activity is 10% of the entire team effort (preparing for the next sprint).
Introducing the concept of "Epic" (large user stories) and "Theme" (collection of user stories) to organize the backlog.
Writing user stories exercise.
Topic is Logging in:
As a registered user, I want to be able to get my existing username and password via email in case I forget it
As a registered user, I want to be able to get my existing username and password via sms in case I forget it
As a registered user, I want to be able to enter my username and password so that I can access the system
As a new I want to be able to register as user in the system
As a new I want to be get an email confirmation with my chosen username and password, when I have successfully registered
Mike: Its ok to write user stories that convays the meaning, but are not "true" (e.g. As a user I want the bank to debit my account when I withdraw money from the ATM)
Arrange User Story Workshops
- Product Owner + team members
Start with Epics and iterate
Why User Stories
- Shift the focus from writing to talking (avoid "you built what I asked for (as written in the spec) but it is not what I need"). This does not imply not writing things down!.. only to include
- Stories are equally understandable by developers and customers
- Stories support and encourage iterative development
- Stories are the right size for planning
- Stories support participatory design - get as close to users as possible
- Stories emphazise the users goals not the system's attributes (example with Car vs Lawn Mover)
Most importantly: Don't forget the purpose. The story text we write on the card is less important than the conversations we have
Q&A
Q: How to incorporate bugs into this process?
A: Some teams pull work from two queues, and then uses Stories to "wrap" bugs.
Consolidating tools is a good thing in this process (e.g. not a separate bug tracking tool from the UserStory tool)
Balance is critical (business and development)
- Technical vs business issues in focus
Resource allocation should be a shared problem (business and development)
Imperfect schedules
- We cannot perfectly predict a software schedule
- If we can't perfectly predict a schedule, we can't say what will be delivered
So what do we do?
- We make decisions based on information we have, but do it often
- We make decisions throughout the whole project
UserStories - Three C's of Ron Jeffries
- Card - The basics of the US
- Conversation - The details, captured in conversations with product owner
- Confirmation - Acceptance tests confirm the story was coded correctly. Defined by Product Owner in cooperation with team
Mike displayed UserStories examples
considers the "so that" clause as optional - use when needed to improve understanding, not when it is obvious
How to capture the detail?
Several options:
1. Write details as "conditions of satisfaction" - these are essentially tests, but we do not use this term (test) since it is misinterpreted by many..
2. Create a hierarchy with the details in sub-user stories
These techniques can and should be combined
Mental model for the product backlog as an iceberg with more detail/smaller items on the top (Sprints), bigger in the middle (Release), and large at the bottom (future releases)
Someone needs to regularly "grooming the product backlog". Estimated required effort for this activity is 10% of the entire team effort (preparing for the next sprint).
Introducing the concept of "Epic" (large user stories) and "Theme" (collection of user stories) to organize the backlog.
Writing user stories exercise.
Topic is Logging in:
As a registered user, I want to be able to get my existing username and password via email in case I forget it
As a registered user, I want to be able to get my existing username and password via sms in case I forget it
As a registered user, I want to be able to enter my username and password so that I can access the system
As a new I want to be able to register as user in the system
As a new I want to be get an email confirmation with my chosen username and password, when I have successfully registered
Mike: Its ok to write user stories that convays the meaning, but are not "true" (e.g. As a user I want the bank to debit my account when I withdraw money from the ATM)
Arrange User Story Workshops
- Product Owner + team members
Start with Epics and iterate
Why User Stories
- Shift the focus from writing to talking (avoid "you built what I asked for (as written in the spec) but it is not what I need"). This does not imply not writing things down!.. only to include
- Stories are equally understandable by developers and customers
- Stories support and encourage iterative development
- Stories are the right size for planning
- Stories support participatory design - get as close to users as possible
- Stories emphazise the users goals not the system's attributes (example with Car vs Lawn Mover)
Most importantly: Don't forget the purpose. The story text we write on the card is less important than the conversations we have
Q&A
Q: How to incorporate bugs into this process?
A: Some teams pull work from two queues, and then uses Stories to "wrap" bugs.
Consolidating tools is a good thing in this process (e.g. not a separate bug tracking tool from the UserStory tool)
Kevlin Henney: Sustainable Software Architecture
Started off with defining the concepts in the title of the talk.
Sustainable - from the Bruntland Report -
Software is applied thought - Alan O'Callaghan
Architecture - Grady Booch - Design vs architecture is related to significance, measured by the cost of change
All systems have an architecture - intentional or accidental
Architect is a role rather than a job
Good architecture reduces the general significance of decisions
An architecture needs to be stable but this is not the same as static
Reflection on the word Agile.
It is something you are, not something you do (Adjective)
Agile architecture -
The question is not if your system has an architecture, but why and how.
Architecture influences your speed. Code will not magically fall into a good structure.
What properties are we looking for in our architecture:
- Solidity, robustness
- Usefulness
- Aesthetic quality - this is the (virtual) work environment for people (developers). It should be pleasant and intentional..
External integrity of the system (that it works according to expectations) is doing the right job
Conceptual integrity - this is a part of doing the job right
Complexities. Two types
- Essential (inherent in the problem domain)
- Accidental complexity involves the complexity found in the solution. Sometimes necessary but often it is noise and introduced by people. (Hello world in EJB..)
The concept of Technical Debt (by Martin Fowler) was discussed
Ok to take shortcuts for tactical reasons, but we always need to undo the shortcut at a later point in time (sooner rather than later). A clear NOTE should be made. If you have to many, you loose grip of them and the significance decreases.. you are on your way into trouble (unmanaged emails in inbox analogy). Use notes on a visual board.
Refactoring is a form of reactive design
We need Feedback to learn. Feedback in all levels (programmer testing, system testing, continuous integrations, architecture retrospectives, demos, code reviews +++)
Focus on collaboration rather than command
- Architects need to be actively involved in development
- Refactoring, testing, collaborative modeling and incremental work needs to be normal activities
- Immersive workshops with a technical focus (not replaced by stand-ups - this is a required addition forgotten by many teams)
Draws a parallel to building architects, they now often make it possible to start using a building long before it is finished (stages) this is certainly also possible with software.
Programming should focus on the continuous present tense (not the future)
Be careful to taking YAGNI too literally, it's a prediction
Sustainable - from the Bruntland Report -
Software is applied thought - Alan O'Callaghan
Architecture - Grady Booch - Design vs architecture is related to significance, measured by the cost of change
All systems have an architecture - intentional or accidental
Architect is a role rather than a job
Good architecture reduces the general significance of decisions
An architecture needs to be stable but this is not the same as static
Reflection on the word Agile.
It is something you are, not something you do (Adjective)
Agile architecture -
The question is not if your system has an architecture, but why and how.
Architecture influences your speed. Code will not magically fall into a good structure.
What properties are we looking for in our architecture:
- Solidity, robustness
- Usefulness
- Aesthetic quality - this is the (virtual) work environment for people (developers). It should be pleasant and intentional..
External integrity of the system (that it works according to expectations) is doing the right job
Conceptual integrity - this is a part of doing the job right
Complexities. Two types
- Essential (inherent in the problem domain)
- Accidental complexity involves the complexity found in the solution. Sometimes necessary but often it is noise and introduced by people. (Hello world in EJB..)
The concept of Technical Debt (by Martin Fowler) was discussed
Ok to take shortcuts for tactical reasons, but we always need to undo the shortcut at a later point in time (sooner rather than later). A clear NOTE should be made. If you have to many, you loose grip of them and the significance decreases.. you are on your way into trouble (unmanaged emails in inbox analogy). Use notes on a visual board.
Refactoring is a form of reactive design
We need Feedback to learn. Feedback in all levels (programmer testing, system testing, continuous integrations, architecture retrospectives, demos, code reviews +++)
Focus on collaboration rather than command
- Architects need to be actively involved in development
- Refactoring, testing, collaborative modeling and incremental work needs to be normal activities
- Immersive workshops with a technical focus (not replaced by stand-ups - this is a required addition forgotten by many teams)
Draws a parallel to building architects, they now often make it possible to start using a building long before it is finished (stages) this is certainly also possible with software.
Programming should focus on the continuous present tense (not the future)
Be careful to taking YAGNI too literally, it's a prediction
18 Jun 2009
Robert C. Martin Clean Practice: Agility and Craftsmanship
Craftmanship is based on diciplines not process.
Work in short cycles
Team and personally
dont wait for definition
- its okay to throw things away, you have learned..
abstract away volatility
Decouple yourself from others (technically and personally). Use Mocking to be able to test your own use of interfaces
Never be blocked (this is one of the most vital ones)
Avoid turgid viscous architectures (cloudy/thick) - e.g. an enterprise architecture created by people that have no responsibility for delivering
Incremental improvement (the boyscout rule) - it will not happen later.. ref example with Sushi chef that cleans his tools as apart of the making..
No grand redesigns - (sidenote: Developers have much power in organisations. The requirements is in the old system.)
- if you have a mess, clean it gradually
- port it in bits to a new technology. Re-implement bit by bit.
Technology and language does not matter.. Good programers
Don't write bad code - there is no way to rush and go fast. Rushing will make you slower.
This is a mental shift when you are under pressure to deliver..
If you want to go fast, go well.. (Programmers put the preassure on themself)
Clean code
TDD
QA should find nothing
Avoid debugging (replaced by writing tests)
Manual Test Scripts are Immoral
Automation is required to do all the trivial regression testing!
The cost of automating is assumed to be 3 manual runs..
Definition of Done
Must not be ambiguous
- Requires successful run of all UnitTests (with 90% coverage)
- Required successful run of all acceptance tests (Uncle Bob uses FitNesse)
Test through the right interface
GUI testing tools test your system through the GUI
Business rules should not be tested via GUI, since the GUI can be unchanged when business rules change and vise versa.
API tests also need to be in place
Databasetests can also be applied when needed
Unit testing (90% coverage)
Components - API tests (50% coverage)
Cross components (20% coverage)
Full system integration (5% coverage)
Exploratory testing via GUI (
Apprenticeship
New people need mentors
Use good tools
Tip: Git source control
Continuous Integration
Use it..
Open Office
Collaborate and interact
Key takeaways: Uncle Bob is persuasive as always. This talk was detailing his keynote on Wednesday.
Work in short cycles
Team and personally
dont wait for definition
- its okay to throw things away, you have learned..
abstract away volatility
Decouple yourself from others (technically and personally). Use Mocking to be able to test your own use of interfaces
Never be blocked (this is one of the most vital ones)
Avoid turgid viscous architectures (cloudy/thick) - e.g. an enterprise architecture created by people that have no responsibility for delivering
Incremental improvement (the boyscout rule) - it will not happen later.. ref example with Sushi chef that cleans his tools as apart of the making..
No grand redesigns - (sidenote: Developers have much power in organisations. The requirements is in the old system.)
- if you have a mess, clean it gradually
- port it in bits to a new technology. Re-implement bit by bit.
Technology and language does not matter.. Good programers
Don't write bad code - there is no way to rush and go fast. Rushing will make you slower.
This is a mental shift when you are under pressure to deliver..
If you want to go fast, go well.. (Programmers put the preassure on themself)
Clean code
TDD
QA should find nothing
Avoid debugging (replaced by writing tests)
Manual Test Scripts are Immoral
Automation is required to do all the trivial regression testing!
The cost of automating is assumed to be 3 manual runs..
Definition of Done
Must not be ambiguous
- Requires successful run of all UnitTests (with 90% coverage)
- Required successful run of all acceptance tests (Uncle Bob uses FitNesse)
Test through the right interface
GUI testing tools test your system through the GUI
Business rules should not be tested via GUI, since the GUI can be unchanged when business rules change and vise versa.
API tests also need to be in place
Databasetests can also be applied when needed
Unit testing (90% coverage)
Components - API tests (50% coverage)
Cross components (20% coverage)
Full system integration (5% coverage)
Exploratory testing via GUI (
Apprenticeship
New people need mentors
Use good tools
Tip: Git source control
Continuous Integration
Use it..
Open Office
Collaborate and interact
Key takeaways: Uncle Bob is persuasive as always. This talk was detailing his keynote on Wednesday.
Rockford Lhotka - Achieving Balance
Topic: How do we deal with the rate of change in technologies?
How can you have mental model to deal with the rate of change without loosing your mind?
Vendors pushes changes in technologies (e.g. WinForms replaced by WPF)
Customers also pushes changes (e.g. why can we not have better analysis capabilities? why does our client software look so bad when my phone looks so cool?, why is changes costing so much money?)
It is our job to find the path to find the path to the perfect result.
The reality is that "perfect" means "good enough".
We have a myriad of choices
- Client/server vs SOA
SOA used to create client/server systems is very expensive
- Smart client vs RIA vs Web
- ADO.NET vs DataSet vs ORM
- Code vs Workflow/modeling
- OO: Data centric vs Behavioural
The Vendors deliver new technologies and "forces" users to keep up the pace and replacing old (but working) technologies.
There are also broader things with potentially bigger impact, e.g. Oslo that predicts a whole new concept of what is involved in developing applications.
Advice: Separate the job from the Task
- Being and architect (hard)
- Architecting software (should not be hard)
Maintaining control:
- Keep the faith
Define the technical vision
Define your technical goals
Define your application goals
Align to the vision and goals
- Keep it simple
Complexity is the enemy
Silver bullets breed complexity
Technologies have a cost - make sure the benefit outweighs the cost
Avoid "golden hammer" anti-pattern
(CAB is mentioned as an example of an "overly complex" solution)
-Plan for the worst, expect the best
"Will technology X really deliver?"
"Will I have a consistent team?"
- You can't get something for nothing
Frameworks have consequences (good and bad)
Patterns have consequences (good and bad)
Architectures have consequences (good and bad)
Do your consequences align with your goals? Always assess cost versus benefit.
- Be a laggard (=someone ho stays behind)
Type A - Bleeding edge - uses beta versions
Type B - Waits for SP1 - upgrades to keep up
Type C - Waits for V3 - upgrades to maintain support
- Don't build frameworks
- Do have an architecture
This is the most critical point..
- that exists in service to vision/goals
- that anticipate future needs
- that transcends application design
- that limits options
- that provides focus and discipline
Remember: Architecture is NOT the same as design. Architecture spans applications.
My takeaway: Interesting topic, but the talk did not give much new information. The issues for architect/architecture is still the same, and no really new insight into how to manage it was given. Nice to know that others struggle with the same issues as I feel myself though :-)
How can you have mental model to deal with the rate of change without loosing your mind?
Vendors pushes changes in technologies (e.g. WinForms replaced by WPF)
Customers also pushes changes (e.g. why can we not have better analysis capabilities? why does our client software look so bad when my phone looks so cool?, why is changes costing so much money?)
It is our job to find the path to find the path to the perfect result.
The reality is that "perfect" means "good enough".
We have a myriad of choices
- Client/server vs SOA
SOA used to create client/server systems is very expensive
- Smart client vs RIA vs Web
- ADO.NET vs DataSet vs ORM
- Code vs Workflow/modeling
- OO: Data centric vs Behavioural
The Vendors deliver new technologies and "forces" users to keep up the pace and replacing old (but working) technologies.
There are also broader things with potentially bigger impact, e.g. Oslo that predicts a whole new concept of what is involved in developing applications.
Advice: Separate the job from the Task
- Being and architect (hard)
- Architecting software (should not be hard)
Maintaining control:
- Keep the faith
Define the technical vision
Define your technical goals
Define your application goals
Align to the vision and goals
- Keep it simple
Complexity is the enemy
Silver bullets breed complexity
Technologies have a cost - make sure the benefit outweighs the cost
Avoid "golden hammer" anti-pattern
(CAB is mentioned as an example of an "overly complex" solution)
-Plan for the worst, expect the best
"Will technology X really deliver?"
"Will I have a consistent team?"
- You can't get something for nothing
Frameworks have consequences (good and bad)
Patterns have consequences (good and bad)
Architectures have consequences (good and bad)
Do your consequences align with your goals? Always assess cost versus benefit.
- Be a laggard (=someone ho stays behind)
Type A - Bleeding edge - uses beta versions
Type B - Waits for SP1 - upgrades to keep up
Type C - Waits for V3 - upgrades to maintain support
- Don't build frameworks
- Do have an architecture
This is the most critical point..
- that exists in service to vision/goals
- that anticipate future needs
- that transcends application design
- that limits options
- that provides focus and discipline
Remember: Architecture is NOT the same as design. Architecture spans applications.
My takeaway: Interesting topic, but the talk did not give much new information. The issues for architect/architecture is still the same, and no really new insight into how to manage it was given. Nice to know that others struggle with the same issues as I feel myself though :-)
Rafal Lukawiecki : Architectual use of Business Intelligence in Application Design
Introducing the idea as BI as a development platform
Data Warehousing and OLAP as application development platform.
Innovative uses of BI for non BI applications
1. Using data warehouse and OLAP in application design
DW for storage?
- Relational DW is an excellent place for all non-transactional data
- Easy queries (star join with optimization)
- MDM for holistic information design
- No need for OLAP
Facts are facts, once written, read-only
Change?= another fact all woth a date stamp (historical versioning)
Use Dimensions - Modern navigation in structures.
Veru compact and beautiful
OLAP - Deductive database (mini)
Use OLAP for logic to de-couple it from the application
Time-partition give support for changing requirements (e.g. calculations) over time. The logic from the pas can always be called.
Embed Predictive KPIs (using DMX and MDX - I have never heard of these terms, some readng to do..)
2. Using data mining in architecture of intelligent applications
What does data minin do
- Finds patterns
- explores your data
- preforms predicions
Examples of intelligent applications.
Business process validation - early detection of failure based on past failures and successes
Adaptive user interface - based on past behaviour
Input validation - based on accepted data, not on fixed rules
This is also known as Predictive programming
What is so special?
Application behaviour evolves and follows its data mining model - influenced by events caused by your appplication. This is AI by the backdoor... :-)
Data mining API's. There are several, but ADO.NET is the most popular.
Need to learn DMX (Data Mining Extensions)
Rafal now had a demo of how to use his for input validation. The app is available on the web (did not get the address..).
The validation logic was driven from the data mining.
What to do:
1. Prepare history (events) for mining
2. Create and rain DM model on your data
3. Test and validate the model
4. The model predicts outcomes
5. Make application logic depend in predicted outcomes
6. Update and re-validate your model periodically
My takeaway: Can be interesting in some cases, but I personnally need much more knowledge in BI before I can evaluate this as an alternative for anything.
Rafal predicts that this will be a standard part of the toolbox used in any application in the future.
Data Warehousing and OLAP as application development platform.
Innovative uses of BI for non BI applications
1. Using data warehouse and OLAP in application design
DW for storage?
- Relational DW is an excellent place for all non-transactional data
- Easy queries (star join with optimization)
- MDM for holistic information design
- No need for OLAP
Facts are facts, once written, read-only
Change?= another fact all woth a date stamp (historical versioning)
Use Dimensions - Modern navigation in structures.
Veru compact and beautiful
OLAP - Deductive database (mini)
Use OLAP for logic to de-couple it from the application
Time-partition give support for changing requirements (e.g. calculations) over time. The logic from the pas can always be called.
Embed Predictive KPIs (using DMX and MDX - I have never heard of these terms, some readng to do..)
2. Using data mining in architecture of intelligent applications
What does data minin do
- Finds patterns
- explores your data
- preforms predicions
Examples of intelligent applications.
Business process validation - early detection of failure based on past failures and successes
Adaptive user interface - based on past behaviour
Input validation - based on accepted data, not on fixed rules
This is also known as Predictive programming
What is so special?
Application behaviour evolves and follows its data mining model - influenced by events caused by your appplication. This is AI by the backdoor... :-)
Data mining API's. There are several, but ADO.NET is the most popular.
Need to learn DMX (Data Mining Extensions)
Rafal now had a demo of how to use his for input validation. The app is available on the web (did not get the address..).
The validation logic was driven from the data mining.
What to do:
1. Prepare history (events) for mining
2. Create and rain DM model on your data
3. Test and validate the model
4. The model predicts outcomes
5. Make application logic depend in predicted outcomes
6. Update and re-validate your model periodically
My takeaway: Can be interesting in some cases, but I personnally need much more knowledge in BI before I can evaluate this as an alternative for anything.
Rafal predicts that this will be a standard part of the toolbox used in any application in the future.
Udi Dahan - Avoid failed SOA
Services
Buisness components
Autonomous components
A 2 year product development project lifecycle compresses into one hour
Business
Archiecture
Technology
"Tenets of service orientation" does not give much practial guidance for developing successful SOA solutions.
Refers to the Microsoft service classification hiearchy as "way up into the ivory tower".
- Process services
- Activity services
- Entity services
Described a "as is" architecture with service to service to service call between modules, that leads to bad performance and issues with technology like the number of threads, memory and locks and how these affect eachother.
The core issue: The synchronous communication = Tight coupling at runtime
Solution: Move to pub/sub
At first, this seems very strange since we have point to point communicaion (service to service calls). There is no one to many in the business process or the solution.
(Udi displayed some fun slides in this section with Yoda"talking" about solutions to our problems, hope to get a copy and re-use this idea)
Pub/sub in practice:
- Let modules subscribe to interesting events from other solutions, act on these events and store the impact for the subscriber locally at the subscriber module
- The events happen over a long period of time
- The process is what emerges from the cascading events, not something you plan
The business wants to have this flexibility - "When this happens (event published) do this (act on subscribed event)"
This is Event Driven architecture - the way the world works.
We realise: We cannot talk about services without talking about the events they publishes.
SOA and EDA are twins..
Minimize duplex request/response.
Services are not about providing services (they don't serve), they just are.
Services, contracts and pub/sub.
The Publisher defines the contract of the Event (message) - what you publish - you own.
Endpoints and buiness are connected.
More than one service in a business component can (and will) subscribe to the same events.
Create autonomous components within, as scale out happens within an autonomous component.
Retorical question: Does this not violate the DRY principle? It seems like we are duplicating logic?
(have to write more about this later)
This architecture allows for applying more infrastructure to targeted parts of the solution, and
Conclusion in the end: This approach complies with the "Tenets of service orientation" and we have real SOA
see more on http://www.UdiDahan.com
My takeaway from this session: This is very interesting stuff, and I really would like to look more into this approach. I think we have major issues in our internal "SOA efforts" and this might be a part of the to-be solution.
Buisness components
Autonomous components
A 2 year product development project lifecycle compresses into one hour
Business
Archiecture
Technology
"Tenets of service orientation" does not give much practial guidance for developing successful SOA solutions.
Refers to the Microsoft service classification hiearchy as "way up into the ivory tower".
- Process services
- Activity services
- Entity services
Described a "as is" architecture with service to service to service call between modules, that leads to bad performance and issues with technology like the number of threads, memory and locks and how these affect eachother.
The core issue: The synchronous communication = Tight coupling at runtime
Solution: Move to pub/sub
At first, this seems very strange since we have point to point communicaion (service to service calls). There is no one to many in the business process or the solution.
(Udi displayed some fun slides in this section with Yoda"talking" about solutions to our problems, hope to get a copy and re-use this idea)
Pub/sub in practice:
- Let modules subscribe to interesting events from other solutions, act on these events and store the impact for the subscriber locally at the subscriber module
- The events happen over a long period of time
- The process is what emerges from the cascading events, not something you plan
The business wants to have this flexibility - "When this happens (event published) do this (act on subscribed event)"
This is Event Driven architecture - the way the world works.
We realise: We cannot talk about services without talking about the events they publishes.
SOA and EDA are twins..
Minimize duplex request/response.
Services are not about providing services (they don't serve), they just are.
Services, contracts and pub/sub.
The Publisher defines the contract of the Event (message) - what you publish - you own.
Endpoints and buiness are connected.
More than one service in a business component can (and will) subscribe to the same events.
Create autonomous components within, as scale out happens within an autonomous component.
Retorical question: Does this not violate the DRY principle? It seems like we are duplicating logic?
(have to write more about this later)
This architecture allows for applying more infrastructure to targeted parts of the solution, and
Conclusion in the end: This approach complies with the "Tenets of service orientation" and we have real SOA
see more on http://www.UdiDahan.com
My takeaway from this session: This is very interesting stuff, and I really would like to look more into this approach. I think we have major issues in our internal "SOA efforts" and this might be a part of the to-be solution.
Roy Osherove: Code Leaders and Beautiful teams
Combining technical practices and leadership
What makes Teams great
- Automation - in all levels. Personal level, team level.
Examples
of "do not use the mouse"
if you do something twice, create a script for it
Get to know your tools/Master your tools
Make people commit to improvement in productivity
TooTips: FinalBuilder
Avoids the unmaintainable mess of MSBuilds, Ant tasks etc.
Test and buy the right tools
Throw out the wrong tools (ClearCase is mentioned as a god example...)
Get quick, continuous feedback
- Does my code work?
- Did I build the right thing? (clarify with customer - every iteration)
- Am I doing things right?
Stand up meetings
Pair programming
Code reviews
Test reviews
Just-in time design decisions
Communicate when you have problems
Create a culture where bringing up issues is a natural thing.
The team (lead) should know it within a day if a developer has problems
Show progress visually and big
This does NOT mean MS Project plans..
TeamLevel: Use whiteboard with notes in columns
- To-Do
- In Process
- Done
This will visualize for everyone the issue of to many items "In progress" at the same time, and productivity is hindered by context switching issues.
http://leansoftwareengineering.com/ksse/scrum-ban/
Limit the amount of tasks assigned to a person at the time (max 2?)
Communicate without "meetings"
Daily standups
Pair programming (research show 120% effort in building, much better quality and communication, less bugs)
Peer code review
Same room
Big visual progress
Create a drawing to visualise the state of the solution - seems strange but have huge impact on common understanding. What are we working on now.
Grow people with integrity
Are the people in my team better this month than he month before? What can I do to challenge and help them?
Make proposals for change and ask people for commitment to change.
Remove obstacles for your team
Daily/weekly questions (from Roys blog)
The talk ended with Roy singing a song (with guitar) on "Knocking on the clients door". Hilarious :-)
What makes Teams great
- Automation - in all levels. Personal level, team level.
Examples
of "do not use the mouse"
if you do something twice, create a script for it
Get to know your tools/Master your tools
Make people commit to improvement in productivity
TooTips: FinalBuilder
Avoids the unmaintainable mess of MSBuilds, Ant tasks etc.
Test and buy the right tools
Throw out the wrong tools (ClearCase is mentioned as a god example...)
Get quick, continuous feedback
- Does my code work?
- Did I build the right thing? (clarify with customer - every iteration)
- Am I doing things right?
Stand up meetings
Pair programming
Code reviews
Test reviews
Just-in time design decisions
Communicate when you have problems
Create a culture where bringing up issues is a natural thing.
The team (lead) should know it within a day if a developer has problems
Show progress visually and big
This does NOT mean MS Project plans..
TeamLevel: Use whiteboard with notes in columns
- To-Do
- In Process
- Done
This will visualize for everyone the issue of to many items "In progress" at the same time, and productivity is hindered by context switching issues.
http://leansoftwareengineering.com/ksse/scrum-ban/
Limit the amount of tasks assigned to a person at the time (max 2?)
Communicate without "meetings"
Daily standups
Pair programming (research show 120% effort in building, much better quality and communication, less bugs)
Peer code review
Same room
Big visual progress
Create a drawing to visualise the state of the solution - seems strange but have huge impact on common understanding. What are we working on now.
Grow people with integrity
Are the people in my team better this month than he month before? What can I do to challenge and help them?
Make proposals for change and ask people for commitment to change.
Remove obstacles for your team
Daily/weekly questions (from Roys blog)
Whole team
- What can we automate?
- where are we "Reinventing the wheel"?
- what are the tools that slow us down?
- what tools can we use better?
- are there bugs that I could have found earlier? how do I make sure I find them earlier?
- when do we find out we built the right thing?
- when do we find out our code\design sucks? how can we make that earlier?
- How do we show progress at the team level? at the management level?
- How many meetings does each dev have every week? how can we remove them?
- Are we building by feature or by layer?
- can we make all our team sit in the same place?
Team Lead
- daily: what bottlenecks exist in the team? what have I solved?
- will my devs be better in a month or two than they were before? if not, how do I make that happen?
- what prevents my devs from working? what am I doing about this?
The talk ended with Roy singing a song (with guitar) on "Knocking on the clients door". Hilarious :-)
Ted Neward: Rethinking "Enterprise"
An almost philosophical talk on "what is the right answer" to a given problem, and how people usually evaluates solutions.
Historical perspective on the what the mentors/teachers/vendors of the software industry try to convince us of, but it is not the same as the truth.
Examples: J2EE/EJB
Why did so many people bite so hard on the J2EE hype wave?
Examples of how the brain works - put things together.
When somebody says "enterprise" your brain looks for things associated with enterprise:
- relational database
- client server type topology
- app servers
Face it, you are prejudiced since you are technacist
So how to "rethink enterprise"?
- Resist the temptation of the familiar
Every project is different
Reject the "goal of reuse" (yagni) until you have written the same system the third time
This is the most prominent cause of failure for projects
- Befriend the Unconformable truth
Be cynical
Question the assumptions
Look for hidden costs
Investigate the implementations
The presenter should also be able to tell you where it should NOT be used
- Eschew the "best practice"
In certain scenarios the "best practice" does more harm than good
Best practice are our attempts to avoid thinking
These practices need a Context to be relevant
Rename to "our current practice"
- Embrace the "perennial gale of creative destruction"
Said about market economies
Same holds for languages, patterns, tools, practices etc.
- Create an evaluation function of your own for each technology/pattern/practice introduced into your awareness
Context matters
- Attend the goals
Define the goals - features and scenarios for runtime behavior expectancies
Conclusion: Rethink the enterprise for yourself!
Historical perspective on the what the mentors/teachers/vendors of the software industry try to convince us of, but it is not the same as the truth.
Examples: J2EE/EJB
Why did so many people bite so hard on the J2EE hype wave?
Examples of how the brain works - put things together.
When somebody says "enterprise" your brain looks for things associated with enterprise:
- relational database
- client server type topology
- app servers
Face it, you are prejudiced since you are technacist
So how to "rethink enterprise"?
- Resist the temptation of the familiar
Every project is different
Reject the "goal of reuse" (yagni) until you have written the same system the third time
This is the most prominent cause of failure for projects
- Befriend the Unconformable truth
Be cynical
Question the assumptions
Look for hidden costs
Investigate the implementations
The presenter should also be able to tell you where it should NOT be used
- Eschew the "best practice"
In certain scenarios the "best practice" does more harm than good
Best practice are our attempts to avoid thinking
These practices need a Context to be relevant
Rename to "our current practice"
- Embrace the "perennial gale of creative destruction"
Said about market economies
Same holds for languages, patterns, tools, practices etc.
- Create an evaluation function of your own for each technology/pattern/practice introduced into your awareness
Context matters
- Attend the goals
Define the goals - features and scenarios for runtime behavior expectancies
Conclusion: Rethink the enterprise for yourself!
17 Jun 2009
The Haacked and Hanselman Show - Website attacks and how to prevent them
Phil Haack and Scott Hanselman did a live "show" where they exploited weaknesses in a website, and then close the "security holes".
Most of it was live demo, and this is some of the attacks they demonstrated:
CSRF attack - Cross Site Resource Forgery (lager et script som poster data (e.g. en kontooverførsel) og lurer bruker til å kjøre dette scriptet (via spam email) mens han er logget inn)
My takeaway: To test and verify that you have a "secure enough" site, requires a big skillset. This is not for every developer, but seems to be something solutions really should do to be "secure enough" according to the paranoia level appropriate for the case at hand.
In my company I know some people that have these skills and I will try to involve them in future web-app "security hardening" activities in our current IT programme.
Most of it was live demo, and this is some of the attacks they demonstrated:
CSRF attack - Cross Site Resource Forgery (lager et script som poster data (e.g. en kontooverførsel) og lurer bruker til å kjøre dette scriptet (via spam email) mens han er logget inn)
My takeaway: To test and verify that you have a "secure enough" site, requires a big skillset. This is not for every developer, but seems to be something solutions really should do to be "secure enough" according to the paranoia level appropriate for the case at hand.
In my company I know some people that have these skills and I will try to involve them in future web-app "security hardening" activities in our current IT programme.
Juval Löwy - Zen of Architecture
An interesting talk on how to define the architecture for a solution.
Quick analysis phase (2-3 days) and then 2-3 days of verification (including stress testing of a vertical). Main point was to capture the run-time capabilities required.
iDesign have a methodology for how to do this, and this talk is a condensed content of "The architects Master class" course held by Juval (hosted by Programutvikling) - in Norway in august.
My computer hung up and had to be restarted during this session, and I lost most of my notes.
Awaiting the presentation to be published..
Quick analysis phase (2-3 days) and then 2-3 days of verification (including stress testing of a vertical). Main point was to capture the run-time capabilities required.
iDesign have a methodology for how to do this, and this talk is a condensed content of "The architects Master class" course held by Juval (hosted by Programutvikling) - in Norway in august.
My computer hung up and had to be restarted during this session, and I lost most of my notes.
Awaiting the presentation to be published..
Udi Dahan Reliability, Availability, and Scalability - How to have your cake, and eat it too
Reliability - How to manage this in a SOA world?
Reliability for the system as a whole is the key to keep smiling.
The core for Adaptability, Reliability and Scalability: Use one way messaging
Reliability
- Don't lose data
- Causing global inconsistency
Scenarios
- App server goes down
- Database goes down
- Database deadlocks (most common)
How does messaging help in not loosing data by accident?
- Introduce a queue before processing
- Include the queue in the transaction
- The rollback (or exception) is detected and rolls back the message on the queue
Calling external webservices
- WS don't roll back (unless you apply WS Atomic Transactions)
- Introduce a messaging gateway between your app and the external web service. Part of the transactions.
Availability
Is it just a question of nines?? (99,99999...)
Response times is also an issue in this context
Command/Response pattern
- What does it mean of I did not get a response?
How does messaging help:
- Moving responsibilities
1. Accept the command - put it in the queue - respond with "I have received the request and will get back to you"
2. Process the request from the queue and respond on a different channel
This introduces changes to the contracts (in WCF : CallbackContracts)
Messaging over HTTP
- Request message - respond with a GUID and a "expected response time"
- Client ask: Is this GUID finished? (if no, repeat as above, if yes, return response)
Query/response interaction
Get response right away
Move to Pub/Sub - We will tell you when a significant change has occured. Allows for caching..
Scalability
Putting data in a queue is a scalable and "simple" approach
- More http servers to put messages in the queue
- More servers to manage queues
- More servers to process messages (pulling) from the queues and provide the responses (on different channels)
This approach allows for different SLA's for different customers/orders. Use different scaling mechanisms.
Issue: Scaling to large messages (e.g. 50MB sized messages)
Solution: Represent the big message as a set of smaller messages
Big messages originate from interactions. Make this protocol explicit. How???
Reliability for the system as a whole is the key to keep smiling.
The core for Adaptability, Reliability and Scalability: Use one way messaging
Reliability
- Don't lose data
- Causing global inconsistency
Scenarios
- App server goes down
- Database goes down
- Database deadlocks (most common)
How does messaging help in not loosing data by accident?
- Introduce a queue before processing
- Include the queue in the transaction
- The rollback (or exception) is detected and rolls back the message on the queue
Calling external webservices
- WS don't roll back (unless you apply WS Atomic Transactions)
- Introduce a messaging gateway between your app and the external web service. Part of the transactions.
Availability
Is it just a question of nines?? (99,99999...)
Response times is also an issue in this context
Command/Response pattern
- What does it mean of I did not get a response?
How does messaging help:
- Moving responsibilities
1. Accept the command - put it in the queue - respond with "I have received the request and will get back to you"
2. Process the request from the queue and respond on a different channel
This introduces changes to the contracts (in WCF : CallbackContracts)
Messaging over HTTP
- Request message - respond with a GUID and a "expected response time"
- Client ask: Is this GUID finished? (if no, repeat as above, if yes, return response)
Query/response interaction
Get response right away
Move to Pub/Sub - We will tell you when a significant change has occured. Allows for caching..
Scalability
Putting data in a queue is a scalable and "simple" approach
- More http servers to put messages in the queue
- More servers to manage queues
- More servers to process messages (pulling) from the queues and provide the responses (on different channels)
This approach allows for different SLA's for different customers/orders. Use different scaling mechanisms.
Issue: Scaling to large messages (e.g. 50MB sized messages)
Solution: Represent the big message as a set of smaller messages
Big messages originate from interactions. Make this protocol explicit. How???
Michael Feathers - Seven Blind Alleys in Software Design
Michael Feathers ObjectMentor
Seven Blind Alleys in Software Design
Are we progressing as an industry?
Learning from experience, see people trying the same things that is proven "it will not work".
This is a talk on the
1. Programming in pictures
Usually does not go all the way and creates maintenance issues for the corner cases
Some complexity is irreducible and cannot be made easy by introducing pictures
Most programming is visual anyway
2. Metaprogramming
Endless flexibility options, but the downside is the understandability.
Need to find the right balance of doing vs overdiong
Examples:
- Can undermine the confidence in the system
- Testing is difficult
3. The ultimate abstraction
Examples: Xanadu - Ted Nelson
Chandler - Mitch Caper
Over-analysis is a keyword.
Most happens when people want ultimate flexibility
And it always seems like it is just a step away..
How to solve: Publicize bad cases
Apply deadlines :-)
4. The perfect taxonomy
Examples: IBM San Fransisco and Taligent.
One Account to rule them all.
Generalizing too early (e.g. frameworks)
The truth: There is no uniform vocabulary across all domains
Languages are ambiguous
5. Modeling the real world
Does not always work. Need software specific abstractions.
6. Code generation
Similar as "programming in pictures".
Should not try to do everything with a code generator.
Over-reliance on code generation can de-skill development.
Cruft buckets - avoiding dealing with complexity. All logic is put in one bucket.
7. Natural languages
The belief that we can program in natural languages.
Fluent interfaces.. examples are date time libraries and jMock?
This is good since it makes things read better, but it is not natural language since it has constraints.
Example: Fluent interfaces, BDD, DSLs,
This is nice, but we need to consider the testability and the constraints that is inevitably present in programming languages
Seven Blind Alleys in Software Design
Are we progressing as an industry?
Learning from experience, see people trying the same things that is proven "it will not work".
This is a talk on the
1. Programming in pictures
Usually does not go all the way and creates maintenance issues for the corner cases
- CASE tools
- Visual Programming Languages
- MDA
- Executable UML
Some complexity is irreducible and cannot be made easy by introducing pictures
Most programming is visual anyway
2. Metaprogramming
Endless flexibility options, but the downside is the understandability.
Need to find the right balance of doing vs overdiong
Examples:
- Smalltalk
- UML Meta metamodel
- Ruby
- Configurable systems
- Can undermine the confidence in the system
- Testing is difficult
3. The ultimate abstraction
Examples: Xanadu - Ted Nelson
Chandler - Mitch Caper
Over-analysis is a keyword.
Most happens when people want ultimate flexibility
And it always seems like it is just a step away..
How to solve: Publicize bad cases
Apply deadlines :-)
4. The perfect taxonomy
Examples: IBM San Fransisco and Taligent.
One Account to rule them all.
Generalizing too early (e.g. frameworks)
The truth: There is no uniform vocabulary across all domains
Languages are ambiguous
5. Modeling the real world
Does not always work. Need software specific abstractions.
6. Code generation
Similar as "programming in pictures".
Should not try to do everything with a code generator.
Over-reliance on code generation can de-skill development.
Cruft buckets - avoiding dealing with complexity. All logic is put in one bucket.
7. Natural languages
The belief that we can program in natural languages.
Fluent interfaces.. examples are date time libraries and jMock?
This is good since it makes things read better, but it is not natural language since it has constraints.
Example: Fluent interfaces, BDD, DSLs,
This is nice, but we need to consider the testability and the constraints that is inevitably present in programming languages
NDC 2009 Kicks off - Robert C. Martin (Uncle Bob): "Are you a professional".
NDC 2009 in Oslo kicks off with a keynote from Robert C. Martin (Uncle Bob) in the topic "Are you a professional".
Started off with a historical perspective and then depicts what he sees as professional behavior.
Principles that defines professional:
1st principle - Do no harm
Function
Examples: make it work and ensure that you validate it, if errors/bugs are discovered in QA, you should ask yourself "how could this happen".
How: Write and run, automated tests. (Coverage in the 90ies) Everyone should know the coverage of their code.
Structure
Do not make a mess, it will slow you down.
Everything you do to code must allow for future changes.
The way to make code changeable is to change it all the time.
Again: Have tests that will validate where you break your code when changing it.
The boyscout rule: When you leave a module, leave it cleaner than it was before.
Test Driven Development: 10 years ago, people where skeptical. It has now been proven that it works well, and must be a part of every developers professionalism.
The 3 laws to TDD.
Avoid debugging, use your tests
Tests works as code examples
Writing test first makes the code testable and more flexible (de-coupled)
Definition of Done - Make this completely clear
- Including unit tests
- Including acceptance tests (written before any code is written)
Lots of tools available
2nd principle - Work ethics
- working overtime should not be the standard
- instead, go home and spend time on evolving your own competence. Experiment, read, learn.
3rd principle - Know your field
- Software development
- Design principles
- Know algorithms
- Design patterns
- Several languages
- Methodologies
4rt principle - Continuous learning
You must learn continuously and it is YOUR responsibility
Step outside your comfort zone - learn new stuff
Do not rely of one source/vendor for learning
5th principle - Practice
Ref. musicians and martial art. This applies to sw development skills as well.
Examples: codingdojo.com
Test-Code pingpong, with clock
6th principle - Collaborate
Pair programming - Learn and collaborate
Look at others when programming, help each other
Avoid silos of information - No single person should be the only one to know a field of interest
7th principles - Mentoring
Mentor juniors and prepare for two way learning
8th principle - Know your domain
Understand more than the requirements, get in a position to ask and challenge them
It is your job to know that you are building the right things
Identify with your customer, make his problems your problems
Understand the needs - not only the requirements
Started off with a historical perspective and then depicts what he sees as professional behavior.
Principles that defines professional:
1st principle - Do no harm
Function
Examples: make it work and ensure that you validate it, if errors/bugs are discovered in QA, you should ask yourself "how could this happen".
How: Write and run, automated tests. (Coverage in the 90ies) Everyone should know the coverage of their code.
Structure
Do not make a mess, it will slow you down.
Everything you do to code must allow for future changes.
The way to make code changeable is to change it all the time.
Again: Have tests that will validate where you break your code when changing it.
The boyscout rule: When you leave a module, leave it cleaner than it was before.
Test Driven Development: 10 years ago, people where skeptical. It has now been proven that it works well, and must be a part of every developers professionalism.
The 3 laws to TDD.
Avoid debugging, use your tests
Tests works as code examples
Writing test first makes the code testable and more flexible (de-coupled)
Definition of Done - Make this completely clear
- Including unit tests
- Including acceptance tests (written before any code is written)
Lots of tools available
2nd principle - Work ethics
- working overtime should not be the standard
- instead, go home and spend time on evolving your own competence. Experiment, read, learn.
3rd principle - Know your field
- Software development
- Design principles
- Know algorithms
- Design patterns
- Several languages
- Methodologies
4rt principle - Continuous learning
You must learn continuously and it is YOUR responsibility
Step outside your comfort zone - learn new stuff
Do not rely of one source/vendor for learning
5th principle - Practice
Ref. musicians and martial art. This applies to sw development skills as well.
Examples: codingdojo.com
Test-Code pingpong, with clock
6th principle - Collaborate
Pair programming - Learn and collaborate
Look at others when programming, help each other
Avoid silos of information - No single person should be the only one to know a field of interest
7th principles - Mentoring
Mentor juniors and prepare for two way learning
8th principle - Know your domain
Understand more than the requirements, get in a position to ask and challenge them
It is your job to know that you are building the right things
Identify with your customer, make his problems your problems
Understand the needs - not only the requirements
Subscribe to:
Posts (Atom)