Industry Trends and Management Principles

Importance of Data

Information – data which is processed and contextualized – helps propel us towards answers and insights. However, we need collaboration –> Someone to lead and make sure we ingest the data, aggregate and store it, and visualize it (for both ourselves and shared in analytics for others). Then, someone to exploit the data. Thus, a culture of collaboration is just as important, as is bringing different skillsets together. It’s not the tools that are the problem, but we need the right organizational structure and the right people (expertise), to both support the data and its usage. Similarly, the problem is usually not the tech, the challenge is changing the culture and processes to allow people to work harder to allow data to inform ourselves, so we can utilize it to drive decisions.

Data Security and Openness

It is important for data to be both secure (protect people) but also shared (allow more researchers to find insights in the data and consequently the public to benefit). It is difficult to focus on both, but completely possible. You address privacy and security at the beginning, assess value and risk, but don’t look at security and data openness as two goals in tension. When data is more secure with less breaches, the more institutions and the people who lead them are willing to be open with their data. It is also important to be transparent with the public of what we are not releasing, being transparent with what we have at the lowest level. Hybrid data warehousing is a good compromise – Production environment in a physical data center, and testing environment in the cloud.
To encourage data to be shared, it is up to the people consuming the data to both prove the data is secure and to prove that it will provide value, because those are the two biggest excuses institutions use for not sharing their data (unsecure and won’t provide value). In addition, regarding research, we must monetize data citations instead of only publications – incentives to not withhold data in order to maintain data rights.
To enable the data to shared, systems should be constructed with Service Oriented Architecture –> have APIs → build a library around them and build a unique user base.

Data Quality and Other Goals/Tools for Development

  • Develop visualizations
  • Be iterative – better to redesign (or re-use materials) 30x to learn from failures than spend everything building one complex design
  • Aim for more data
  • Include the confidence level of the data, data should be highly structured and organized
  • Understand what type of analysis of your data will benefit the public the most – Descriptive (describing the data) vs Predictive (predicting future data, inductive reasoning) vs Prescriptive (predicting outcomes of decisions). Ideal method of predictive and prescriptive is usually inductive reasoning (starting with the data and creating a theory/hypothesis based on it) vs deductive (starting with a hypothesis and coming to a conclusion).

Machine Learning and Deep Learning

Programming – In functional programming, all functions have inputs and outputs. Higher order programming is where the functions are also inputs and outputs.
Machine Learning is programming where you use examples instead of explicit instructions. Called “Software 2.0”. Example: Pandora finds patterns nuanced enough to make a prediction of what music you like.
Deep Learning is a type of machine learning where we focus on the internal implementation of how a system is learning, where there are layers of intermediate representations. In other words, the data is transformed at different levels. Deep learning is not a completely separate field from machine learning – it’s just a method that works better than others.

  • Linear Progression
  • Clustering Algorithms
  • Neural nets – a type of learning that involve back propagation. However, it is not a good word because it is not like copying the brain with math formulas. It is more like a child learning – for example, a child first learns the difference between a human and an animal. But it still doesn’t know the difference between animals, then he learns that 4 legged creatures are usually animals, but still doesn’t know the difference between horses and dogs. Iterative learning. Good for image recognition and time series and audio (in general, recurring things).

    Real world examples

    • Working with DARPA and Air Force: Instead of staring at the pixels at the screen, be told what’s on the ground shooting at you without looking.
    • Visual field guide for soldiers on the ground – Something bit me – is it poisonous? Should I go back to base or stay out?
    • Agtech – Find where water is saturated and fertilizer is needed.
    • Fighting poachers and tracking animals
    • Surveillance footage – instead of human eyes watching them (error prone and expensive), neural net can tell you there was 2 people possibly outside of your house – please review the tapes at the time period.

Project Management Process and Dynamics

The Process
David Kantor -> Structure determines performance like a riverbed determines the flow of river
1) Set needs and goals (gather requirements first)
2) Diagnose – do research, set standards, define roles and responsibilities. Talk to lead to figure out how to prioritize and manage people’s expectations of delivery.
3) Develop stakeholder lists, training, metrics, processes, frameworks
4) Implement the idea and execute

Teamwork Principles
The best teams are both hierarchical and collaborative –> Collaborative for brainstorming and bringing together ideas, knowledge sharing and team efforts. Hierarchical for when it’s time to start converging, when everyone has their role to play in providing an answer for a client.

Other Principles
Think about things systematically. Always look at the why’s.
Encourage documentation – it is important not to lose any decisions or insights made in meetings, or lose any industry / process knowledge due to turnover
“Make it ugly” – have meetings saying what’s wrong – no delay in talking about issues and figuring out how to fix them

Stages of Cultural Evolution
(from less to greater complexity)
1) Conformist/Role-Driven – following authority, stability, consistency
2) Achievement/Results-Driven – innovation, results, competition, meritocracy
3) Pluralistic/Relationship-Driven
4) Evolutionary/Purpose-Driven

Three Kinds of Mental Complexity in Adults
(from less to greater complexity)
1) (30% of individuals) Pre-Modern Simple Reasoning – Step-by-step instructions.
2) (50% of individuals) Modern Mindset – Value high-IQ, scientific method, domain knowledge.
2) (20% of individuals) Post-Modern – Want to see different perspectives and understand the whole system.

Problems for Agility Today
History of Organizational Landscape – A fixed process used to be pure innovation at the time, like how agile is cutting-edge right now. Companies, however, made up of more processes, required to have greater structure and greater size.
Problem – The landscape most organizations face now is inherently complex, which requires an upward shift in mental complexity of organizational leaders (as an OS needs to get more complex to install more complex applications), and greater agility in leadership. And the issue is, most managers/leaders see complex systems as though they are complicated. But instead, an Agile Leader should have the ability to breakdown systems logically and explain them, in addition to the following skills:
Outer Skills – Ability to influence others, evoke shared sense making (how we see the world), skills in facilitating/catalyzing, creatively apply domain knowledge, systems thinking skills.
Inner Skills – Complex meaning making, high emotional intelligence (self awareness / management), clear about and grounded in a sense of purpose, permeable to the perspectives of others (values – what we care about) and self identity (how we see ourselves).

A consultant helps the customer succeed over their issues, challenges, or problems they may have. The reason consultants charge a lot of money is because they need to be right all the time, tackling the difficult problems quickly and at a high level of quality that the client either does not have the skill or resources to do. Consultants work on the critical business data and provide insights the client has trouble uncovering. In order to do this, the consultant must be more receptive to the client than even their employees, be able to learn quickly the intricate knowledge regarding the client’s service and the problem(s) the client faces.

Overview of Software Languages and Software Tools

Cloud Servers: AWS
Programming Languages: Python, Apache Spark
Machine Learning: Scikit for Python, Pyflux for Python
Deep Learning: Caffe
Neural Networks: Tensor Flow