Tag: MongoDB

NoSQL Design Considerations and Lessons Learned

Posted by on July 29, 2015

At Rivet Logic, we’ve always been big believers and adopters of NoSQL database technologies such as MongoDB. Now, leading organizations worldwide are using these technologies to create data-driven solutions to help them gain valuable insight into their business and customers. However, selecting a new technology can turn into an over engineered process of check boxes and tradeoffs. In a recent webinar, we shared our experiences, thought processes and lessons learned building apps on NoSQL databases.

The Database Debate

The database debate is never ending, where each type of database has its own pros and cons. Amongst the multitude of databases, some of the top technologies we’ve seen out in the marketing include:

  1. MongoDB – Document database
  2. Neo4j – Graph based relationship
  3. Riak – Key value data store
  4. Cassandra – Wide column database

Thinking Non-Relational

When it comes to NoSQL databases, it’s important to think non-relational. With NoSQL databases, there’s no SQL query language or joins. It also doesn’t serve as a drop-in replacement for Relational Databases, as they are two completely different approaches to storing and accessing data.

Another key component to consider is normalized vs. denormalized data. Whereas data is normalized in relational databases, it’s not a necessity or important design consideration for NoSQL databases. In addition, you can’t use the same tools, although that’s improving and technology companies are heavily investing in making their tools integrate with various database technologies. Lastly, you need to understand your data access patterns, and what it looks like from the application level down to the DB.

Expectations

Also keep in mind your expectations and make sure they’re realistic. Whereas the Relational model is over 30 years old, the NoSQL model is much younger at approximately 7 years, and enterprise adoption occurring within the last 5 years. Given the differences in maturity, NoSQL tools aren’t going to have the same level of maturity as those of Relational DB’s.

When evaluating new DB technologies, you need to understand the tradeoffs and what you’re willing to give up – whether it be data consistency, availability, or other features core to the DB – and determine if the benefits outweigh the tradeoffs. And all these DB’s aren’t created equally – they’re built off of different models for data store and access, use different language – which all require a ramp up.

In addition, keep in mind that scale and speed are all relative to your needs. Understanding all of these factors in the front end will help you make the right decision for the near and long term.

Questions to Ask Yourself

If you’re trying to determine if NoSQL would be a good fit for a new application you’re designing, here are some questions to ask yourself:

  1. Will the requirements evolve? Most likely they will, rarely are all requirements provided upfront.
  2. Do I understand the tradeoffs? Understand your must have vs. like to have.
  3. What are the expectations of the data and patterns? Read vs. write, and how you handle analytics (understand operational vs. analytics DB and where the overlap is)
  4. Build vs. Buy behavior? Understand what you’re working with internally and that changing internal culture is a process
  5. Is the ops team on board? When introducing new DB technologies, it’s much easier when the ops team is on board to make sure the tools are properly optimized.

Schema Design Tidbits

Schema is one the most critical things to understand when designing applications for these new databases. Ultimately the data access patterns should drive your design. We’ll use MongoDB and Cassandra as examples as they’re leading NoSQL databases with different models.

When designing your schema for MongoDB, it’s important to balance your app needs, performance and data retrieval. Your schema doesn’t have to be defined day 1, which is a benefit of MongoDB’s flexible schema. MongoDB also contains collections, which are similar to tables in relational DB’s, where documents are stored. However, the collections don’t enforce structure. In addition, you have the option of embedding data within a document, which depending on your use case, could be highly recommended.

Another technology to think about is Cassandra, a wide column database where you model around your queries. By understanding the access patterns, and the types of questions your users are asking the DB, then you can design your schema to be more accurate. You also want to distribute data evenly across nodes. Lastly, you want to minimize partition (groups of rows that share the same key) reads.

Architecture Examples

MongoDB has a primary-secondary architecture, where the secondary would become the primary if it ever failed, resulting in the notion of never having a DB offline. There are also rights, consistency, and durability, with primaries replicating to the secondaries. So in this model, the database is always available, where data is consistent and replicated across nodes, all performed in the backend by MongoDB. In terms of scalability, you’re scaling horizontally, with nodes being added as you go, which introduces a new concept of sharding, involving how data dynamically scales as the app grows.

On the other hand, Cassandra has a ring-based architecture, where data is distributed across nodes, similar to MongoDB’s sharding. There are similar patterns, but implemented differently within technologies. The diagram below illustrates architectural examples of MongoDB and Cassandra. All of these can be distributed globally, with dynamic scalability, the benefit being you can add nodes effortlessly as you grow.

NoSQL Data Solution Examples

Some of the NoSQL solutions we’ve recently built include:

Data Hub (aka 360 view, omni-channel) – A collection of various data sources pooled into a central location (in this case we used MongoDB), where use cases are built around the data. This enables new business units to access data they might not previously have access to, empowering them to build new products, understand how other teams operate, and ultimately lead to new revenue generating opportunities and improved processes across the organization

User Generated Content (UGC) & Analytics – Storing UGC sessions (e.g. blog comments and shares) that need to be stored and analyzed in the backend. A lot of times the Document model makes sense for this type of solution. However, as technologists continue to increase their NoSQL skill sets, there’s going to be an increasing amount of overlap of similar uses cases being built across various NoSQL DB types.

User Data Management – Also known as Profile Management, and storing information about the user, what they recently viewed, products bought, etc. With a Document model, the flexibility really becomes powerful to evolve the application as you can add attributes as you go, without the need to have all requirements defined out of the gate.

Lessons Learned

When talking about successful deployments, some of the lessons learned we’ve noticed include:

  1. Schema design is an ongoing process – From a Data Hub perspective, defining that “golden record” is not always necessary, as long as you define consistent fields that can be applied everywhere.
  2. Optimization is a team effort – It’s not just the developer’s job to optimize the schema, just like it’s not just the Ops team’s job to make sure the DB is always on. NoSQL is going to give you tunability across these, and the best performance and results
  3. Test your shard keys (MongoDB) – If sharding is a new concept for you, make sure you do your homework, understand and validate with someone that knows the DB very well.
  4. Don’t skimp on testing and use production data – Don’t always assume that the outcome is going to be the same in production.
  5. Shared resources will impact performance – Keep in mind if you’re deploying in the cloud that shared resources will impact distributed systems. This is where working with your Ops team will really help and eliminate frustrations.
  6. Understand what tools are available and where they are in maturity – Don’t assume existing tools (reporting, security, monitoring, etc.) will work in the same capacity as with Relational DB’s, and understand the maturity of the integration.
  7. Don’t get lost in the hype – Do your homework.
  8. Enable the “data consumer” – Enable the person that’s going to interact with the DB (e.g. data analyst) to make them comfortable working with the data.
  9. JSON is beautiful

To summarize, education will eliminate hesitation, and don’t get lost in the marketing fluff. Get Ops involved, the earlier and more often you work with your Ops team, the easier and more successful your application and your experience with these technologies will be. Lastly, keep in mind that these are just DB tools, so you’ll still need to build a front end.

Click here to see a recording of the webinar.

Click here for the webinar slides.

Vitalize Your Content in 2015

Posted by on January 13, 2015

revitalize-conceptAt the beginning of every year, the web is flooded with blog posts, articles, and infographics with predictions and trends of what’s in store for the year ahead.

This year, there are a few key trends that seem to consistently appear in every prediction, and they all seem to revolve around mobile, social, personalization/targeting, and analytics.

Not surprisingly, with mobile on an unrelenting rise, organizations large and small are shifting towards a mobile first strategy. And as we’re surrounded by more and more digital content, organizations need to find creative ways to grab users’ attentions, through delivery of targeted and personalized content, and with social features that encourage audience participation.

In a recently published white paper, Vitalizing Content to Drive Social Customer Engagement, we introduce the concept of Vital Content, a new content class born from connecting content, which ultimately enables a deeper level of user engagement.

Traditional Social Enablement Limitations

In this age of the customer, consumers expect their online experiences to be seamless and omni-channel, filled with consistent and contextual data, all the while engaging them through bi-directional conversations.

Traditionally social content and social enablement has been handled with a collection of individual platforms, perhaps one for reviews, another for discussion forums, yet another for ratings and so on. Having content stuck in such silos limits the value we can expect to derive and deliver from our social platforms. While traditional platforms have helped facilitate conversations and drive greater engagement with customers, these individual channels can often seem unrelated and disjoint.

 “Vital Content” and Its Production Challenges

Motivating engagement and participation in the content lifecycle establishes a lasting and valuable relationship with your customers. To build this kind of deep relationship with your customers you must give them a voice and provide them with content and functionality that is vital to their needs. The answer can be found in a combination of process and technology designed to personalize the experience, gather insight, and surface connected content.

This process produces a new content class — Vital Content — resulting from content creators and consumers building a deeper relationship as each learns more about the other. The outcome of this process keeps users actively engaged, connected longer, and produces a more meaningful experience.

However, traditional solutions fail to build an ongoing relationship with the audience because they fail to keep the right content in front of the right people and encourage engagement that breathes new life into the content. Users today want, and expect, a personalized experience that is consistent and contextually relevant and that spans across their entire customer journey. They shouldn’t have to re-educate at each engagement event on their likes, dislikes or previous history. Instead, they should be presented with relevant content that addresses their needs and triggers new engagement. The process of building a relationship with your user or customer is ongoing, and technology should enable that relationship to prosper.

Building Relationships Through Metadata

So how is this accomplished? Since content, comments, ratings and other social content are essentially the same, by connecting them with metadata, it’s possible to build relationships between them, pulling them out of their traditional silos. Through the application of metadata such as tagging, content curators and end users are able to create relationships between any piece of content or commentary, regardless of the source. These cross-referenced pieces can then be dynamically embedded, restructured and linked together in endless configurations.

With these ends in mind, Crafter Software has created Crafter Social, an innovative platform leveraging MongoDB, for creating Vital Content to help organizations maximize their customer engagement and the strength of their customer relationships. Crafter Social enables an increased level of engagement with the user while enhancing the overall experience. Furthermore, requirements will evolve as the user’s engagement increases over time. Crafter Social provides a flexible approach built on a system of relationships, and as these relationships grow, it provides the tools to take action on new data types and sources.

Click here to read more and download the full white paper.

Building Engaging Customer Experiences Powered by MongoDB

Posted by on July 08, 2014

This spring and summer, the MongoDB Road Show stops in over 20 cities across the country to educate users on how MongoDB can be used to build modern business apps to improve the customer experience and accelerate time to market. Rivet Logic sponsored several of the cities and presented on the topic of building engaging customer experiences with MongoDB, discussing how a modern database can be used to better leverage existing data to derive business value. The next MongoDB Road Show is this Thursday, July 10, in San Francisco!

What Organizations Need

Organizations seeking to build engaging customer experiences on the Web often have a similar set of goals. To start, they want to increase user adoption by providing an engaging experience that brings value to the end-user. This can lead to increased customer retention, allowing organizations to create loyal customers who can then become their own brand ambassadors.

Moreover, organizations want to capitalize on their customers’ and users’ creativity and innovation by seamlessly weaving in the ability to collaborate, interact, and share into every aspect of the user experience. Businesses find the quality of this type of engagement to be particularly beneficial, due to its unpredictability. However, to enhance the value of these interactions, users need a motivator, meaning organizations need to create high quality content that’s personalized and targeted to each user’s needs.

While personas are great and have worked well to capture general types of users, in reality, users think of themselves more as individuals, with evolving interests over time. Organizations are now faced with delivering personalized experiences beyond a persona level and at an individual level.

What’s the Problem?

However, many organizations are having a hard time with this fine-grained personalization, and it’s largely due to the limiting technology they’re working with. IT teams are often faced with seemingly “impractical” features that business teams are requesting.

Organizations today are using separate systems like standalone content apps – blogs, forums, wikis, – commenting engines, traditional databases, and BI tools to enable user interaction and collect and analyze information about them. The quality of user interactions is largely driven by the quality of the user-generated content being collected and analyzed. However, since much of this valuable customer data is silo’d in disparate systems, it’s not allowing businesses to effectively leverage their existing data.

While many have attempted to find workarounds for this, there hasn’t been any real success in creating a coherent rich user interaction data set that brings value to all the delivery use cases available. For example, when a user joins the comment thread of a blog entry, they are unaware of the possibility of a forum thread that is discussing the same topic. In addition, these solutions are typically backed by traditional databases, which requires changing of the infrastructure to accommodate new use cases, posing a challenge.

The fact is, the various types of interactions that exist today are disjointed, resulting in redundancy and little chance of connecting and leveraging them. It’s critical that we make these interactions context-aware, and the only way of effectively doing so is to have a holistic view of all the user-generated content that’s being collected, while also allowing the interactions to cross application boundaries.

Pillars of a Good Solution

Successful solutions that meet these challenges must adhere to the following pillars:

Flexibility – The solution must be implemented using technology agnostic building blocks. Being a certain type of shop (.Net, PHP, Windows, etc.) constrains organizations from using the right tools for the job. Using technology agnostic building blocks as the underlying infrastructure allows organizations to innovate and improve their business without being held back by technology.

Scalability – The solution must be scalable without sacrificing performance. There are many platforms out there that claim to be scalable, but what good is that when scaling means long page load times?

Visibility – It’s also extremely important to be able to know and see the overall picture and have a holistic view of user interactions that isn’t so low-level where it prevents you from seeing what they are doing (as is the case with auditing services).

Insight – Lastly, when you have rich, contextual data available in one place, organizations need to be able to leverage that information, innovate and provide new features, capability, and value to their customers.

Case Study – AT&T Developer Community

Now let’s take a look at how a solution like this might be used in the real world. AT&T is currently undergoing an initiative to build a solution to enhance the user experience of their developer community site. The existing site’s collaboration tools are traditional in nature (i.e. blogs and forums), where user engagement is fragmented, making it difficult to find interesting content and reducing collaboration value.

To resolve this, Rivet Logic is implementing a solution that enables user-generated content to cross application boundaries and reside in one location via Crafter Social, while also allowing for better personalization by using Crafter Profile to maintain a dynamic customer profile.

Crafter Social easily adds social engagement features – user-generated comments, likes, ratings, blogs, discussion forums, and more – to a website by attaching social features to any content item or page. And Crafter Profile provides user profile and account management to help create personalized experiences.

For example, in the current site, if a user comments on a blog entry and another user participates in a forum discussion about the same topic, these interactions are not associated in any way.

With the Crafter Social solution, we were able to turn the blog entry’s comment thread into a virtual forum, thus connecting the two threads of discussion into one. This simple approach is extremely powerful, satisfying all four pillars of a good solution focused on enhancing customer engagement.

Even more, due to Crafter Social’s flexible architecture and underlying data model, it can easily be extended into other use cases, made possible by MongoDB’s document-based data models. In addition, the ability to easily embed Crafter Social into any site using any technology makes it an ideal part of any developer’s toolkit.

As illustrated in the diagram below, Crafter Social is broken into two parts. On the client side, it can be embedded on any site page regardless of what technology was used for implementation. And on the server side, Crafter Social collects various data from different sites and use cases, maintaining a holistic view of the user data. All of this helps enhance the quality of business intelligence information generated.

With this solution, AT&T is able to achieve their goals of increased user adoption and enhanced user engagement and retention. MongoDB plays a key role in the solution’s success by enabling:

  • Flexibility – Create new apps without revisiting infrastructure
  • Scalability – Ability to store large amounts of data and query without hurting performance
  • Visibility – Data is structured in an intuitive way allowing easy translation from raw data to something actionable
  • Insight – Flexible data structures and queries pave the way for creativity and innovation

To download a copy of the slides, click here.