Tag: big data

Find Meaning in Your Data With Elasticsearch

Posted by on March 28, 2017

We’re surrounded by data everywhere we go, and the amount is growing with each action we take. We rely on it regularly, probably a lot more than we even realize or would like to admit. From searching for nearby restaurants while traveling, to reading online product reviews prior to making a purchase, and finding the best route home based on real-time traffic patterns, data helps us make informed decisions every day.

However, all that data on its own is just data. The real value comes when you can find the right, relevant data when it’s needed. Better yet, take it a step further and find meaning in your data, and that’s where the real goldmine is.

Businesses are increasingly turning to search and analytics solutions to derive more value from their data, helping to provide the deep insights necessary to make better business decisions. Some popular use cases include:

  • Intelligent Search Experiences to discover and deliver relevant content
  • Security Analytics to better understand your infrastructure’s security
  • Data Visualization to present your data in a meaningful way
  • Log Analytics to gain deeper operational insight

At Rivet Logic, we realize the importance of data, and see the challenges businesses are facing in trying to make sense of their growing data pools. We’re excited to have partnered with Elastic – the company behind a suite of popular open source projects including ElasticsearchKibanaBeats, and Logstash – to deliver intelligent search and analytics solutions to help our customers get the most value out of their data, allowing them to make actionable improvements to websites for enhanced customer experiences!

A Real-world Use Case

How might this apply in a real-world scenario, you ask?

An example is a global hospitality customer of ours, who has partnered with Rivet Logic to implement three internal facing web properties that enable the company to perform its day to day business operations. With a reach spanning across 110+ countries, these sites are deployed in the cloud on Amazon AWS throughout the US, Europe and Asia Pacific, consisting of many data sources and used across multiple devices.

This customer needed a way to gain deeper insight into these systems — how the sites are being used along with the types of issues encountered to help improve operational efficiencies. Using Elasticsearch and Kibana, this customer is able to gain much better visibility into each site’s utility. Through detailed metrics, combined with the ability to perform aggregations and more intelligent queries, this customer can now gain much deeper insight into their data set through in depth reports and dashboards. In addition, the Elastic Stack solution aggregates all system logs into one place, making it possible to perform complex analysis to provide insightful data to better address operational concerns.

 

NoSQL Design Considerations and Lessons Learned

Posted by on July 29, 2015

At Rivet Logic, we’ve always been big believers and adopters of NoSQL database technologies such as MongoDB. Now, leading organizations worldwide are using these technologies to create data-driven solutions to help them gain valuable insight into their business and customers. However, selecting a new technology can turn into an over engineered process of check boxes and tradeoffs. In a recent webinar, we shared our experiences, thought processes and lessons learned building apps on NoSQL databases.

The Database Debate

The database debate is never ending, where each type of database has its own pros and cons. Amongst the multitude of databases, some of the top technologies we’ve seen out in the marketing include:

  1. MongoDB – Document database
  2. Neo4j – Graph based relationship
  3. Riak – Key value data store
  4. Cassandra – Wide column database

Thinking Non-Relational

When it comes to NoSQL databases, it’s important to think non-relational. With NoSQL databases, there’s no SQL query language or joins. It also doesn’t serve as a drop-in replacement for Relational Databases, as they are two completely different approaches to storing and accessing data.

Another key component to consider is normalized vs. denormalized data. Whereas data is normalized in relational databases, it’s not a necessity or important design consideration for NoSQL databases. In addition, you can’t use the same tools, although that’s improving and technology companies are heavily investing in making their tools integrate with various database technologies. Lastly, you need to understand your data access patterns, and what it looks like from the application level down to the DB.

Expectations

Also keep in mind your expectations and make sure they’re realistic. Whereas the Relational model is over 30 years old, the NoSQL model is much younger at approximately 7 years, and enterprise adoption occurring within the last 5 years. Given the differences in maturity, NoSQL tools aren’t going to have the same level of maturity as those of Relational DB’s.

When evaluating new DB technologies, you need to understand the tradeoffs and what you’re willing to give up – whether it be data consistency, availability, or other features core to the DB – and determine if the benefits outweigh the tradeoffs. And all these DB’s aren’t created equally – they’re built off of different models for data store and access, use different language – which all require a ramp up.

In addition, keep in mind that scale and speed are all relative to your needs. Understanding all of these factors in the front end will help you make the right decision for the near and long term.

Questions to Ask Yourself

If you’re trying to determine if NoSQL would be a good fit for a new application you’re designing, here are some questions to ask yourself:

  1. Will the requirements evolve? Most likely they will, rarely are all requirements provided upfront.
  2. Do I understand the tradeoffs? Understand your must have vs. like to have.
  3. What are the expectations of the data and patterns? Read vs. write, and how you handle analytics (understand operational vs. analytics DB and where the overlap is)
  4. Build vs. Buy behavior? Understand what you’re working with internally and that changing internal culture is a process
  5. Is the ops team on board? When introducing new DB technologies, it’s much easier when the ops team is on board to make sure the tools are properly optimized.

Schema Design Tidbits

Schema is one the most critical things to understand when designing applications for these new databases. Ultimately the data access patterns should drive your design. We’ll use MongoDB and Cassandra as examples as they’re leading NoSQL databases with different models.

When designing your schema for MongoDB, it’s important to balance your app needs, performance and data retrieval. Your schema doesn’t have to be defined day 1, which is a benefit of MongoDB’s flexible schema. MongoDB also contains collections, which are similar to tables in relational DB’s, where documents are stored. However, the collections don’t enforce structure. In addition, you have the option of embedding data within a document, which depending on your use case, could be highly recommended.

Another technology to think about is Cassandra, a wide column database where you model around your queries. By understanding the access patterns, and the types of questions your users are asking the DB, then you can design your schema to be more accurate. You also want to distribute data evenly across nodes. Lastly, you want to minimize partition (groups of rows that share the same key) reads.

Architecture Examples

MongoDB has a primary-secondary architecture, where the secondary would become the primary if it ever failed, resulting in the notion of never having a DB offline. There are also rights, consistency, and durability, with primaries replicating to the secondaries. So in this model, the database is always available, where data is consistent and replicated across nodes, all performed in the backend by MongoDB. In terms of scalability, you’re scaling horizontally, with nodes being added as you go, which introduces a new concept of sharding, involving how data dynamically scales as the app grows.

On the other hand, Cassandra has a ring-based architecture, where data is distributed across nodes, similar to MongoDB’s sharding. There are similar patterns, but implemented differently within technologies. The diagram below illustrates architectural examples of MongoDB and Cassandra. All of these can be distributed globally, with dynamic scalability, the benefit being you can add nodes effortlessly as you grow.

NoSQL Data Solution Examples

Some of the NoSQL solutions we’ve recently built include:

Data Hub (aka 360 view, omni-channel) – A collection of various data sources pooled into a central location (in this case we used MongoDB), where use cases are built around the data. This enables new business units to access data they might not previously have access to, empowering them to build new products, understand how other teams operate, and ultimately lead to new revenue generating opportunities and improved processes across the organization

User Generated Content (UGC) & Analytics – Storing UGC sessions (e.g. blog comments and shares) that need to be stored and analyzed in the backend. A lot of times the Document model makes sense for this type of solution. However, as technologists continue to increase their NoSQL skill sets, there’s going to be an increasing amount of overlap of similar uses cases being built across various NoSQL DB types.

User Data Management – Also known as Profile Management, and storing information about the user, what they recently viewed, products bought, etc. With a Document model, the flexibility really becomes powerful to evolve the application as you can add attributes as you go, without the need to have all requirements defined out of the gate.

Lessons Learned

When talking about successful deployments, some of the lessons learned we’ve noticed include:

  1. Schema design is an ongoing process – From a Data Hub perspective, defining that “golden record” is not always necessary, as long as you define consistent fields that can be applied everywhere.
  2. Optimization is a team effort – It’s not just the developer’s job to optimize the schema, just like it’s not just the Ops team’s job to make sure the DB is always on. NoSQL is going to give you tunability across these, and the best performance and results
  3. Test your shard keys (MongoDB) – If sharding is a new concept for you, make sure you do your homework, understand and validate with someone that knows the DB very well.
  4. Don’t skimp on testing and use production data – Don’t always assume that the outcome is going to be the same in production.
  5. Shared resources will impact performance – Keep in mind if you’re deploying in the cloud that shared resources will impact distributed systems. This is where working with your Ops team will really help and eliminate frustrations.
  6. Understand what tools are available and where they are in maturity – Don’t assume existing tools (reporting, security, monitoring, etc.) will work in the same capacity as with Relational DB’s, and understand the maturity of the integration.
  7. Don’t get lost in the hype – Do your homework.
  8. Enable the “data consumer” – Enable the person that’s going to interact with the DB (e.g. data analyst) to make them comfortable working with the data.
  9. JSON is beautiful

To summarize, education will eliminate hesitation, and don’t get lost in the marketing fluff. Get Ops involved, the earlier and more often you work with your Ops team, the easier and more successful your application and your experience with these technologies will be. Lastly, keep in mind that these are just DB tools, so you’ll still need to build a front end.

Click here to see a recording of the webinar.

Click here for the webinar slides.

Building Engaging Customer Experiences Powered by MongoDB

Posted by on July 08, 2014

This spring and summer, the MongoDB Road Show stops in over 20 cities across the country to educate users on how MongoDB can be used to build modern business apps to improve the customer experience and accelerate time to market. Rivet Logic sponsored several of the cities and presented on the topic of building engaging customer experiences with MongoDB, discussing how a modern database can be used to better leverage existing data to derive business value. The next MongoDB Road Show is this Thursday, July 10, in San Francisco!

What Organizations Need

Organizations seeking to build engaging customer experiences on the Web often have a similar set of goals. To start, they want to increase user adoption by providing an engaging experience that brings value to the end-user. This can lead to increased customer retention, allowing organizations to create loyal customers who can then become their own brand ambassadors.

Moreover, organizations want to capitalize on their customers’ and users’ creativity and innovation by seamlessly weaving in the ability to collaborate, interact, and share into every aspect of the user experience. Businesses find the quality of this type of engagement to be particularly beneficial, due to its unpredictability. However, to enhance the value of these interactions, users need a motivator, meaning organizations need to create high quality content that’s personalized and targeted to each user’s needs.

While personas are great and have worked well to capture general types of users, in reality, users think of themselves more as individuals, with evolving interests over time. Organizations are now faced with delivering personalized experiences beyond a persona level and at an individual level.

What’s the Problem?

However, many organizations are having a hard time with this fine-grained personalization, and it’s largely due to the limiting technology they’re working with. IT teams are often faced with seemingly “impractical” features that business teams are requesting.

Organizations today are using separate systems like standalone content apps – blogs, forums, wikis, – commenting engines, traditional databases, and BI tools to enable user interaction and collect and analyze information about them. The quality of user interactions is largely driven by the quality of the user-generated content being collected and analyzed. However, since much of this valuable customer data is silo’d in disparate systems, it’s not allowing businesses to effectively leverage their existing data.

While many have attempted to find workarounds for this, there hasn’t been any real success in creating a coherent rich user interaction data set that brings value to all the delivery use cases available. For example, when a user joins the comment thread of a blog entry, they are unaware of the possibility of a forum thread that is discussing the same topic. In addition, these solutions are typically backed by traditional databases, which requires changing of the infrastructure to accommodate new use cases, posing a challenge.

The fact is, the various types of interactions that exist today are disjointed, resulting in redundancy and little chance of connecting and leveraging them. It’s critical that we make these interactions context-aware, and the only way of effectively doing so is to have a holistic view of all the user-generated content that’s being collected, while also allowing the interactions to cross application boundaries.

Pillars of a Good Solution

Successful solutions that meet these challenges must adhere to the following pillars:

Flexibility – The solution must be implemented using technology agnostic building blocks. Being a certain type of shop (.Net, PHP, Windows, etc.) constrains organizations from using the right tools for the job. Using technology agnostic building blocks as the underlying infrastructure allows organizations to innovate and improve their business without being held back by technology.

Scalability – The solution must be scalable without sacrificing performance. There are many platforms out there that claim to be scalable, but what good is that when scaling means long page load times?

Visibility – It’s also extremely important to be able to know and see the overall picture and have a holistic view of user interactions that isn’t so low-level where it prevents you from seeing what they are doing (as is the case with auditing services).

Insight – Lastly, when you have rich, contextual data available in one place, organizations need to be able to leverage that information, innovate and provide new features, capability, and value to their customers.

Case Study – AT&T Developer Community

Now let’s take a look at how a solution like this might be used in the real world. AT&T is currently undergoing an initiative to build a solution to enhance the user experience of their developer community site. The existing site’s collaboration tools are traditional in nature (i.e. blogs and forums), where user engagement is fragmented, making it difficult to find interesting content and reducing collaboration value.

To resolve this, Rivet Logic is implementing a solution that enables user-generated content to cross application boundaries and reside in one location via Crafter Social, while also allowing for better personalization by using Crafter Profile to maintain a dynamic customer profile.

Crafter Social easily adds social engagement features – user-generated comments, likes, ratings, blogs, discussion forums, and more – to a website by attaching social features to any content item or page. And Crafter Profile provides user profile and account management to help create personalized experiences.

For example, in the current site, if a user comments on a blog entry and another user participates in a forum discussion about the same topic, these interactions are not associated in any way.

With the Crafter Social solution, we were able to turn the blog entry’s comment thread into a virtual forum, thus connecting the two threads of discussion into one. This simple approach is extremely powerful, satisfying all four pillars of a good solution focused on enhancing customer engagement.

Even more, due to Crafter Social’s flexible architecture and underlying data model, it can easily be extended into other use cases, made possible by MongoDB’s document-based data models. In addition, the ability to easily embed Crafter Social into any site using any technology makes it an ideal part of any developer’s toolkit.

As illustrated in the diagram below, Crafter Social is broken into two parts. On the client side, it can be embedded on any site page regardless of what technology was used for implementation. And on the server side, Crafter Social collects various data from different sites and use cases, maintaining a holistic view of the user data. All of this helps enhance the quality of business intelligence information generated.

With this solution, AT&T is able to achieve their goals of increased user adoption and enhanced user engagement and retention. MongoDB plays a key role in the solution’s success by enabling:

  • Flexibility – Create new apps without revisiting infrastructure
  • Scalability – Ability to store large amounts of data and query without hurting performance
  • Visibility – Data is structured in an intuitive way allowing easy translation from raw data to something actionable
  • Insight – Flexible data structures and queries pave the way for creativity and innovation

To download a copy of the slides, click here.

What’s in Store for Digital Experience Management in 2014

Posted by on January 13, 2014

2014 is here in full swing, and promises to be an exciting year as the web continues to evolve and new products and trends continue to disrupt the industry.

In 2013, we saw the continued rise of mobile and the age of the customer, where enterprises worldwide re-evaluated strategies to optimally engage with their customers in this digital era. As mobile devices continue to proliferate and have become the new norm, consumers have increasingly higher expectations of the right content delivered to them when and how they want it, resulting in customer experience management skyrocketing to the top of every organization’s priority list.

However, experience management doesn’t just apply to customers, but instead extends to include all organizational stakeholders – customers, employees, partners, etc. We also saw organizations take a closer look internally and focus their efforts on employee community building. Realizing that workplace environments are changing, with a greater need for enterprise collaboration now than ever before, companies are implementing social intranet solutions that offer dynamic and social environments to facilitate community and collaboration.

In addition, many organizations are also building web-based social communities for their external stakeholders for further engagement to improve customer relations and build brand loyalty.

Tackling these daunting digital experience management tasks require careful planning and execution. Organizations need to first determine their business strategies and goals and take the time to really understand their audience to formulate the right messaging. A well thought out strategy sets the right foundation to build your systems – customer experience management, social intranet, customer portal, etc. – upon. The technology should be an enabler of your goals and facilitate your business users to effectively carry out your business objectives.

At Rivet Logic, we believe that software should be agile systems that can easily be customized to fit each organization’s unique needs. There’s no one-size-fits-all tool, and your underlying system must be flexible and developer friendly to allow various customizations and integrations with other existing enterprise applications. In addition, your system must be user friendly for business users. As we’ve seen over the past few years, there’s been a shift from IT to Marketing as Marketing’s responsibilities have expanded to include multi-channel web content management, customer experience management, and more. The tools we employ must be easy to use for non-technical business users.

In 2014 we’ll continue to see these trends evolve. Organizations will continue to put a large emphasis on customer experience management and creating a seamless omni-channel experience as mobile continues to grow. Businesses will also focus more on big data. The explosive growth of social media and mobile devices has generated an enormous amount of user behavioral data that can be harnessed to provide organizations with valuable insight on how to better address the needs of their customers and employees.