Tag: Best practices

NoSQL Design Considerations and Lessons Learned

Posted by on July 29, 2015

At Rivet Logic, we’ve always been big believers and adopters of NoSQL database technologies such as MongoDB. Now, leading organizations worldwide are using these technologies to create data-driven solutions to help them gain valuable insight into their business and customers. However, selecting a new technology can turn into an over engineered process of check boxes and tradeoffs. In a recent webinar, we shared our experiences, thought processes and lessons learned building apps on NoSQL databases.

The Database Debate

The database debate is never ending, where each type of database has its own pros and cons. Amongst the multitude of databases, some of the top technologies we’ve seen out in the marketing include:

  1. MongoDB – Document database
  2. Neo4j – Graph based relationship
  3. Riak – Key value data store
  4. Cassandra – Wide column database

Thinking Non-Relational

When it comes to NoSQL databases, it’s important to think non-relational. With NoSQL databases, there’s no SQL query language or joins. It also doesn’t serve as a drop-in replacement for Relational Databases, as they are two completely different approaches to storing and accessing data.

Another key component to consider is normalized vs. denormalized data. Whereas data is normalized in relational databases, it’s not a necessity or important design consideration for NoSQL databases. In addition, you can’t use the same tools, although that’s improving and technology companies are heavily investing in making their tools integrate with various database technologies. Lastly, you need to understand your data access patterns, and what it looks like from the application level down to the DB.

Expectations

Also keep in mind your expectations and make sure they’re realistic. Whereas the Relational model is over 30 years old, the NoSQL model is much younger at approximately 7 years, and enterprise adoption occurring within the last 5 years. Given the differences in maturity, NoSQL tools aren’t going to have the same level of maturity as those of Relational DB’s.

When evaluating new DB technologies, you need to understand the tradeoffs and what you’re willing to give up – whether it be data consistency, availability, or other features core to the DB – and determine if the benefits outweigh the tradeoffs. And all these DB’s aren’t created equally – they’re built off of different models for data store and access, use different language – which all require a ramp up.

In addition, keep in mind that scale and speed are all relative to your needs. Understanding all of these factors in the front end will help you make the right decision for the near and long term.

Questions to Ask Yourself

If you’re trying to determine if NoSQL would be a good fit for a new application you’re designing, here are some questions to ask yourself:

  1. Will the requirements evolve? Most likely they will, rarely are all requirements provided upfront.
  2. Do I understand the tradeoffs? Understand your must have vs. like to have.
  3. What are the expectations of the data and patterns? Read vs. write, and how you handle analytics (understand operational vs. analytics DB and where the overlap is)
  4. Build vs. Buy behavior? Understand what you’re working with internally and that changing internal culture is a process
  5. Is the ops team on board? When introducing new DB technologies, it’s much easier when the ops team is on board to make sure the tools are properly optimized.

Schema Design Tidbits

Schema is one the most critical things to understand when designing applications for these new databases. Ultimately the data access patterns should drive your design. We’ll use MongoDB and Cassandra as examples as they’re leading NoSQL databases with different models.

When designing your schema for MongoDB, it’s important to balance your app needs, performance and data retrieval. Your schema doesn’t have to be defined day 1, which is a benefit of MongoDB’s flexible schema. MongoDB also contains collections, which are similar to tables in relational DB’s, where documents are stored. However, the collections don’t enforce structure. In addition, you have the option of embedding data within a document, which depending on your use case, could be highly recommended.

Another technology to think about is Cassandra, a wide column database where you model around your queries. By understanding the access patterns, and the types of questions your users are asking the DB, then you can design your schema to be more accurate. You also want to distribute data evenly across nodes. Lastly, you want to minimize partition (groups of rows that share the same key) reads.

Architecture Examples

MongoDB has a primary-secondary architecture, where the secondary would become the primary if it ever failed, resulting in the notion of never having a DB offline. There are also rights, consistency, and durability, with primaries replicating to the secondaries. So in this model, the database is always available, where data is consistent and replicated across nodes, all performed in the backend by MongoDB. In terms of scalability, you’re scaling horizontally, with nodes being added as you go, which introduces a new concept of sharding, involving how data dynamically scales as the app grows.

On the other hand, Cassandra has a ring-based architecture, where data is distributed across nodes, similar to MongoDB’s sharding. There are similar patterns, but implemented differently within technologies. The diagram below illustrates architectural examples of MongoDB and Cassandra. All of these can be distributed globally, with dynamic scalability, the benefit being you can add nodes effortlessly as you grow.

NoSQL Data Solution Examples

Some of the NoSQL solutions we’ve recently built include:

Data Hub (aka 360 view, omni-channel) – A collection of various data sources pooled into a central location (in this case we used MongoDB), where use cases are built around the data. This enables new business units to access data they might not previously have access to, empowering them to build new products, understand how other teams operate, and ultimately lead to new revenue generating opportunities and improved processes across the organization

User Generated Content (UGC) & Analytics – Storing UGC sessions (e.g. blog comments and shares) that need to be stored and analyzed in the backend. A lot of times the Document model makes sense for this type of solution. However, as technologists continue to increase their NoSQL skill sets, there’s going to be an increasing amount of overlap of similar uses cases being built across various NoSQL DB types.

User Data Management – Also known as Profile Management, and storing information about the user, what they recently viewed, products bought, etc. With a Document model, the flexibility really becomes powerful to evolve the application as you can add attributes as you go, without the need to have all requirements defined out of the gate.

Lessons Learned

When talking about successful deployments, some of the lessons learned we’ve noticed include:

  1. Schema design is an ongoing process – From a Data Hub perspective, defining that “golden record” is not always necessary, as long as you define consistent fields that can be applied everywhere.
  2. Optimization is a team effort – It’s not just the developer’s job to optimize the schema, just like it’s not just the Ops team’s job to make sure the DB is always on. NoSQL is going to give you tunability across these, and the best performance and results
  3. Test your shard keys (MongoDB) – If sharding is a new concept for you, make sure you do your homework, understand and validate with someone that knows the DB very well.
  4. Don’t skimp on testing and use production data – Don’t always assume that the outcome is going to be the same in production.
  5. Shared resources will impact performance – Keep in mind if you’re deploying in the cloud that shared resources will impact distributed systems. This is where working with your Ops team will really help and eliminate frustrations.
  6. Understand what tools are available and where they are in maturity – Don’t assume existing tools (reporting, security, monitoring, etc.) will work in the same capacity as with Relational DB’s, and understand the maturity of the integration.
  7. Don’t get lost in the hype – Do your homework.
  8. Enable the “data consumer” – Enable the person that’s going to interact with the DB (e.g. data analyst) to make them comfortable working with the data.
  9. JSON is beautiful

To summarize, education will eliminate hesitation, and don’t get lost in the marketing fluff. Get Ops involved, the earlier and more often you work with your Ops team, the easier and more successful your application and your experience with these technologies will be. Lastly, keep in mind that these are just DB tools, so you’ll still need to build a front end.

Click here to see a recording of the webinar.

Click here for the webinar slides.

Enabling Smarter Enterprise Collaboration Through Social Intranets

Posted by on November 19, 2013

Traditional intranets – while offering a variety of tools for improving internal communication, workforce productivity, collaboration, and more – are often seen as stuffy, boring systems that employees NEED to use, but not necessarily would WANT to use.

However, that’s all changing now. In an age where Customer Experience Management (CEM/CXM) and producing engaging digital experiences across various channels is at the top of every organization’s digital strategy list, some of that is crossing over to modern intranet solutions. Organizations are now beginning to see the importance of providing that same level of engagement to their employees that they do for their customers.

A new generation of social intranets have emerged that focus on building internal employee communities while incorporating social tools to facilitate knowledge sharing, employee interaction and feedback, and team-building and collective problem solving.

Over the last few months, we’ve presented on this hot topic at several events, including two webinars, two Liferay Roadshows, the Liferay North America Symposium, and KMWorld Conference. Let’s take a deeper look at how a social intranet solution can be used to facilitate smarter enterprise collaboration.

Social Communication is Critical – From Internet to Intranet

Looking at the history of social media in general is important in helping us understand why social features have become an essential part of today’s intranets. Most of us were around before social media and remember when it all first started. As with any new technology, there was some initial resistance. However, it was interesting to see how quickly people naturally gravitated towards using social media and embraced the openness that came with it.

Fast-forward to today and you see users worldwide that are socially active. This proves that the need for social interaction is not cultural, but rather it fulfills a natural human need – the need to share, to discover new things, and to be connected to one another. This is why social media has become an essential part of our internet experience today.

Naturally, companies have caught on to this, and Web applications and services that provide social features started sprouting up globally. In fact, it’s hard to find a successful Web application nowadays that doesn’t incorporate social features to some extent.

However, for a while, social interactions were kept out of the workplace, and employees would change their behaviors to fit within the constraints imposed by their employers. And those who were initially rejected for rallying for social interaction within the workplace were the same visionaries that saw the benefits – as a tool for providing constructive feedback, and to help employees be more efficient, hence improving productivity.

This initial resistance to social interaction within the workplace was no different from the resistance from internet users when social media was first conceived. Intranets are changing, and organizations leading that change are already reaping the benefits.

The Social Intranet

So what makes an intranet social? Unlike the public internet, intranets are trickier, since there are usually a lot of organizational rules to follow, many of which go against the essence of being social. Here we will cover some common features of a social intranet.

Intuitive

First and foremost, it has to be easy to use. Just like social applications on the internet, social intranets shouldn’t require user training. Nowadays, people learn how to use Web tools by talking to each other about it. While the availability of online help resources is always useful, discussion forums teach users a lot more.

Light Community Management

One of the biggest obstacles that get in the way of social intranet adoption are an organization’s existing anti-social rules, such as policing all content. Imagine if every time you posted a forum question someone had to review and approve it before it gets posted. Chances are, you’d never ask anything. Instead of being policed, social intranets should be moderated, where users can post freely, and processes can be put in place to correctly handle improper posts. This also means that some organizations would need to change their company culture and rules prior to adopting a social intranet.

Content is Produced and Consumed

Another key difference between social internet and social intranet apps is that consumers of content are as important as producers of content. We tend to always praise producers and call consumers leeches. This isn’t the case in social intranets, however, since it’s the consumers who are the ones using this content to be more efficient and productive at work. Producers without consumers are useless.

Integrated Search

And last but not least, social intranets tend to be heavy on enterprise search. Discovery is an essential social feature, and search is one of the best tools for enabling discovery, so great social intranets typically put a lot of emphasis on integration with enterprise-wide search solutions.

The change is here, and early adopters are reaping the benefits. In fact, a recent research report published by Nielson Norman Group on social intranets has found that many companies see intranet information sharing and other social features offering true competitive advantages. And more interestingly, many executives recognize that social tools are an expected part of a knowledge worker’s standard toolkit.

Building a Social Intranet – Now What?

Building a successful social intranet means selecting the right tools for the job, and Liferay Portal is one of the best tools to take on this task. Liferay provides all the social intranet features discussed in two ways – through Liferay Social Office or as a tailored social intranet built on the Liferay platform.

Liferay Social Office

Liferay Social Office is a packaged social collaboration solution that successfully addresses many social intranet features.

  • Dashboard – A dashboard is an essential part of any social intranet and offers a great way to keep users informed of what’s going on.
  • Sites – Liferay has site-based architecture, allowing organizations to create sites that serve very different purposes – departmental sites, project and team sites, and even regular websites with editorial content.
  • Contact Center – A big part of social communication is to stay connected, so social intranets put a lot of emphasis on having an easy-to-use people directory. Liferay successfully addresses this through Contact Center, which even includes user profiles in search results.
  • Team Calendar – Liferay’s calendar feature can be used in any site to manage site-specific events.
  • Search – Liferay enables enterprise search through integrations with Apache Solr, Google Search Appliance, Endeca, and more, allowing you to make enterprise-wide information discoverable from within the social intranet.
  • Liferay Sync – Liferay addresses the growing need for ubiquitous file sharing through Liferay Sync, a feature similar to Dropbox/Google Drive. In addition, your enterprise security constraints are honored even when you’re outside of your company’s firewall.
  • Related Content – This feature allows content authors to associate content of different types, resulting in a very rich and dynamic information architecture. For example, reader of a blog post who then posts a relevant question in a forum can relate the blog and forum posts, exposing future blog post readers to the forum posts.

Even more, Liferay Social Office has additional social features such as micro-blogging, private messaging, announcements and alerts, full-featured document management, comments and ratings, and much more.

Custom Social Intranet Solution

Another approach to building a Liferay-based social intranet is to create a custom solution utilizing Liferay as a platform. One of the most popular Liferay use cases is actually a Human Resources (HR) Portal, which is often the first step towards building a social intranet. An HR Portal provides an efficient means of disseminating company information and news, and offers an ideal medium for employee outreach and engagement.

Over the past few years, Rivet Logic has built a variety of intranet solutions for customers with varying business models. Through working with these customers and understanding their requirements for these intranets, we’ve found certain features to be commonly sought across all intranet solutions. Based on these features, we’ve built an open source HR Portal that we’ve contributed back to the Liferay community, now available through the Liferay Marketplace. This easy-to-deploy portal solution comes with many useful features, including:

  • Corporate news authoring and publishing
  • Customizable news publishing channels
  • User-friendly people directory
  • Customizable portal-wide main navigation bar
  • Smart news carousel
  • Customizable quick links

Choosing the Right Approach

So how do organizations determine which approach to use for their social intranet? Liferay Social Office is Liferay’s equivalent to Sharepoint, so it may be better suited for organizations that like what they’re getting out of the box and aren’t looking for a tailored solution.

Those looking for a more tailored solution can either use the HR Portal as a starting point and build an intranet on top of it, or use Liferay as a platform and build an intranet from scratch. The bottom line is, there is no right or wrong approach, but rather determining what each organization needs and finding the approach that best fits with those requirements.

Learn More!

If you’d like to learn more about social intranets, we have several resources to help you gain a better understanding:

Boston area Alfresco user group meeting on November 17th to cover Alfresco Best Practices

Posted by on November 05, 2009

Join us in Waltham MA (Greater Boston area) and learn best practices for approaching and implementing projects with Alfresco ECM suite

  • Determining your business use case
  • Architectural best practices
  • Development best practices
  • Operational best practices

Our presentation was co-authored by Russ Danner of Rivet Logic, Jeff Potts of Optaros and Peter Monks of Alfresco

Where?
Rivet Logic Developer Forge, Boston
1050 Winter St
Waltham , MA
617.834.2781

When: 6:30 PM

Pizza, coffee and soda will be provided.

RSVP here to attend