Reading JUnit Source Code

@xiaojun-zhang

Preparation and JUnit Basic Usage

First step is to download the JUnit source code, create a new project in Eclipse, and then import the downloaded JUnit code.

Second step is to define the code to be tested. Below is the code, it’s very simple, just calculates the sum of two integers.

public class SimpleClass {
    public static int add(int a, int b) {
        return a + b;
    }
}

Third step is to write the test case. JUnit test cases can be implemented by inheriting the junit.framework.TestCase class. Methods, whose name begins with ‘test’, defined in subclasses of ‘TestCase’ will be treated as test cases. Multiple test cases can form a TestSuite. JUnit4 can also define test cases with @Test annotation.

public class TestSimpleClass extends TestCase {
    public void testAdd() {
        int result = SimpleClass.add(1, 1);
        Assert.assertEquals(2, result);
    }
}

In the above code, the testAdd() method is parsed as a test case. You can also define a test case by defining the suite() method, which isn’t too much difference.

Now we are ready to run and test the code.

JUnit Source Code Structure

Below is JUnit’s package structure:

Package Stucture

The core code is in junit.framework package. Package junit.runner is for running the test cases, in which BaseTestRunner is an abstract class. Package Junit.awtui, junit.swingui, junit.textui are three different graphical interfaces to run test cases, each package has a TestRunner inherited from junit.runner.BaseTestRunner.

The code I red is mainly in junit.framework package, junit.textui package, and the abstract class BaseTestRunner.

Below is the class diagram:

Class Diagram

Running Process of JUnit

JUnit’s running process can be divided into three steps, preparaing test cases, running test cases and collecting test results. In following code, getTest(testcase) is to prepare test cases, doRun(suite, wait) is to run the test cases:

try {
    if (!method.equlas(""))
        return runSingleMethod(testCase, method, wait);
        
    Test suite = getTest(testCase);
    return doRun(suite, wait);
} catch (Exception e) {
    throw new Exception("Could not create and run test suite: " + e);
}

Below is the sequence diagram:

Sequence Diagram

Design Pattern in JUnit

Composite

JUnit uses composite pattern when creating test cases. Junit.framework.Test is the interface that defines the test case, which includes both run() and countTestCases () methods. TestCase and TestSuite are two classes that implement the interface, TestCase is for single test case, and TestSuite is for a collection of test cases.

Observer

Observer pattern is used when collecting test results. junit.framework.TestResult manges a collection of TestListeners, these TestLiteners will be notified when execution of test case is failed.

Manage a collection of TestListener:

/**
* Registers a TestListener
*/
public synchronized void addListener(TestListenser listener) {
    fListeners.addElement(listener);
}

/**
*  unregisters a TestListener
*/
public synchronized void removeListener(TestListenser listener) {
    fListeners.removeElement(listener);
}

When the test case fails to execute, notify TestListenner to handle the errors:

public synchronized void addError(Test test, Throwable t) {
    fErrors.addElement(new TestFailure(test, t));
    for (Enumeration e = cloneListeners().elements; e.hasMoreElements; ) {
        ((TestListener)e.nextElement()).addError(test, t);
    }
}

Template Method

Template mode is relatively simple, it’s used in BaseTestRunner.getTest(), TestCase.runBare().

Link to article on GitHub

Implementing FAQ Bots using Microsoft Bot Framework

 Subitha Sudhakaran

When we build complex apps, the customers or users will have several queries regarding the working of this App. In normal case, we will provide help documents, FAQ or a forum to submit their queries etc. But now the trend is to move to conversational computing. Bots are the emerging mode for conversational computing. We can use Bots to simplify the customer interactions using the services like Skype, mail, Facebook messenger etc. We can consider it as an artificial user. Even if it is an artificial user, it should be smart enough to do the things which replace human activities. But Bots can be smart, semi- smart or dummy based on how we implement it. Artificial intelligence is the key for Smart BOTs.

I have been exploring and working on using Microsoft Bot framework at SequoiaAT, we can build and deploy chat bot’s across different services. It is intended to be cross- platform and cloud-based. Each Bot application is available to the users through a set of channels such as Skype, mail, text etc. Bot connector is the central component that connected our Bot with these channels. You can configure in which all channels your bot should be available. The architecture of a Bot connector is REST based i.e. there is no constraint on framework or the language. The only constraints are given by the endpoint names and exposed behaviour. Bot builder SDK s an open source SDK hosted on Github. There are two SDKs, one for to use with .Net and one for Node.js.

FAQ BOTS – Microsoft is continuing to update their services, adding new features and functions. Now it includes support for Azure Bot services, a cloud based development platform. Microsoft has implemented different templates to make it easier to use. One of the useful templates is “Question and Answer”. This is an ideal template for anyone building a customer service bot because it can work with your FAQs. In many cases, question and answer already exists in contents like FAQ in some documents. QnAMaker tool will allows us to consume the existing FAQ content and expose it as an HTTP endpoint. QnAMaker is a free web based service to respond to the users’ queries in a conversational way. So there is no coding required developing, training and managing the Bot service using QnA Maker.  As an end user you are only required to provide the predefined question and answer to the QnA  and It takes care of the rest. The result will be an endpoint which accepts the questions and returns a json response containing the matching response.

Bot framework is implemented by wiring up with LUIS (Language Understanding Intelligent Serivce). We don’t need to ask the exact question we stored in knowledgebase to get the relevant answer. It will look for keywords and based on the keywords, it provide appropriate responses. Sometimes it may match more than one possible responses. In that case, it will return the answer with high score. QnA Maker will help us to take our existing knowledge base and help us to turn into conversational services.

Pre-requisite for creating a FAQ BOT using the QnA Maker is a Microsoft id to create a QnA Bot service. We can login to the site https://qnamaker.ai/ for creating the QnA Service. Creating the QnA service is a pretty straight forward with minimum steps. One of the advantages I see here is, it is easy to create a FAQ bot for existing applications if we already have FAQ documents in some document format or in deployed in any page. Supported FAQ documents are .tsv, .pdf, .doc, .docx. So basically we can provide this document or page URL as an input while creating the QnA service. We are permitted to provide multiple input files and it will add the question and answer as a key value pair in its knowledge base. For new applications, if we don’t have any FAQ documents already, we create Question answer pair while creating the service itself. It supports testing our service in a conversational way before publishing the service. Once we published the service, we can make simple HTTP calls to call the QnA Maker service from our BOT applications. Basically when a user ask a question, the BOT application will internally pass the user keyword to the QnA service and will check the knowledge base for approximate matching record.

Once you are done creating the Bot application and integrating it with the QnA service, we need to publish that on the Microsoft Azure. For that you need Microsoft Azure subscription. You can get a free trial for limited period. It is possible to publish our Bot application directly from the Visual studio using the Publish option available. We just only need to follow the Publish wizard steps to complete the process. Finally we can see our published application on Microsoft Azure. After publishing, we need to register your Bot with Microsoft Bot framework.

For registering your Bot, you need to sign in https://dev.botframework.com/ using your Microsoft id and click on the register bot option. We need to fill in the details it needs. There we can see a BOT ID and an App Secret .These are used for authenticating our Bot application with the Microsoft Bot framework. Bot interacts with users naturally from your website or from different other channels. Current supported channels are Skype, Slack, Facebook Messenger, Office 365 mail, teams etc. We can decide in which on channel we would like to add our Bot application while registering the BOT. We can select the channels from the list of supported channels. Once you have added your bot to any of these services, it is always online and you can send message anytime and get an instant response.

Get ready to talk with Smart Devices

Aju Kuriakose

Controlling devices at home and workplace with voice was what we saw only in SciFi movies a few years ago. However with the advancement in AI, NLP etc. this has become a reality now. The number of devices with which we can interact via voice at home and office has grown in past 3-4 years.

Amazon has an undisputed leadership in this and its ecosystem is way ahead in market penetration compared to its competitor. There are many more companies in the foray including OK google and Cortana from Microsoft. There are also many smaller companies like Cyberon, Conexant all working towards enabling voice controlled devices.

As per Strategy Analysis voice could capture up to 12% of Industrial-IoT applications by 2022 and In the consumer segment, voice has the potential to capture up to 18% of application in the 2020 to 2022 timeframe.

Using voice to control devices will dominate and become the most preferred way to interact with IoT devices in coming years. It is because voice is the most natural way to communicate. Amazon and Google have made it very easy to voice enable smart devices. It is also very affordable today as devices don’t need to add much processing power because they leverage the cloud infrastructure to do the heavy lifting. Many companies are going to leverage this model, but there are companies like sensory which is trying to push the voice enablement to the edge. Its TrulyNatural is an embedded large vocabulary continuous speech recognizer system for devices which may not be cloud connected like a toaster or a space heater or a coffee maker.

Device manufacturers will have to pick bets on which technologies to integrate into their product for voice control as the landscape is still evolving. Most likely companies will end up integrating 2-3 leading voice enablement API’s into their product lines.

The key factors to be considered while making the technology choice is (a) Signal to noise ratio. These smart devices will be used in a variety of environment and the ability to capture voice by reducing background sounds is very important (b) Identification & Isolation – The system should have the capability to separate command from other ambient sounds. (c) Capture – Most times the source could be moving or at distance from the mic ( 1 meter to 10 meters) based on the environment in which it is installed. The devices should have the capability to accommodate this.

However, there are privacy and security issues which will have to be addressed as the popularity grows because voice-enabled devices are always listening for the wake-up keywords. So if someone hacks into the system, they will be able to listen to your private conversations. Also, ethical usage of the information obtained becomes important because companies trying to build on their NLP and AI algorithms may decide to listen to all our conversations to strengthen their capabilities.

Interacting with devices via voice reduces multiple steps in our daily lives when we control them. ( Example controlling a thermostat at home, in past we had to get up, go to the thermostat and press buttons multiple times to set the desired temperature. Today we can do it just with a simple statement ” Set temperature to 72 Degrees.”) . So irrespective of the challenges it may bring, we will continue to expand the boundaries of voice capability to control devices. As speech changed humans forever, enabling voice-based commands to communicate with everyday devices will change the world forever.  

Link to article on Linkedin

Configuring headless systems

Ram Mohan Ramakrishnan

Ram Mohan Ramakrishnan

Headless systems abound in the embedded world, in consumer electronics, industrial automation, communications, automobiles etc. In these embedded devices, a Human Machine Interface (HMI) is conspicuous by its absence. That’s to say the device lacks a user display like a monitor, or LCD panel and nor does it have an input device like a keyboard/mouse or remote control.

Yet many of these systems need manual configuration by the user. The authorized user should be able to change certain operating parameters of the device within a predefined set of values. For instance, the Right/Left speaker settings in a home audio system, or safety thresholds in an industrial chemical reaction process, or DNS Server setting on a network router, or an engine speed/RPM limit setting on a telematics device etc.

User configuration circumvents the need to change the embedded software code (also known as the firmware) to address each and every individual user’s preference/scenarios. This would otherwise imply access to the source code (which in most cases is not open to users), making the required code modification, followed by compilation and re-flashing the updated firmware into the device. Configurability, the attribute that allows users to change key system parameters (that are internally used for computation by the firmware code) to predefined values at run-time, has traditionally been a critical design consideration for embedded system designers.

The user performs configuration settings usually using a software application that runs on a desktop or mobile device that presents the user with a configuration interface. The first generation of configuration applications (configuration files being their predecessors) ran on the desktop and used the serial/RS232 port, omnipresent at that time, to connect to the embedded device. These Apps were ported later on either to USB, which replaced RS232 on most modern computers, or ran as such by leveraging the virtual com port (a USB driver abstraction) that made the USB port appear as a normal serial port to the App. Examples of these applications can be found with printers, CNC machines, process control equipment etc.

After the World Wide Web established itself as a platform-independent paradigm for server access, the HTTP protocol came to be adopted across the spectrum of embedded designers for the purpose of device configuration. The inventors of the “www” may never have dreamed that the advantage they had created of not having to install a client application (and ending up managing multiple versions of client Apps for various OS/versions) would some day result in the creation of tiny embedded web-servers that run inside embedded devices. These are low footprint HTTP servers (like HTTPD) which make it possible for the embedded device to be configured using just a browser running on any platform. The IP connectivity typically being over either wired network like Ethernet, or wireless that is Wi-Fi or cellular (GPRS/3g), implemented as light-weight TCP/IP protocol stacks (a la lwIP) to run on resource constrained systems.

For instance, most telecom equipment, network switches, routers and gateways come with configuration pages that can be accessed using a browser. The first step in configuration typically is login, to authenticate user credentials and confirming the level of authority to perform configuration tasks. Subsequently a set of configuration pages are served out from a central menu, where the user can set/change specific parameter values for a group of configuration parameters related to IP networking, that are grouped together on that page. For enterprise routers, the configuration includes simple settings, like DHCP/Static IP, Default Gateway, Subnet mask, DNS Settings etc., but grow pretty complex, to Firewall/DMZ settings, Access Control, VPN/tunneling, Port forwarding etc. Thus it requires a qualified and experienced networking engineer to configure it correctly and to maintain the network equipment up and running.

Narrowing down focus to the world of consumer electronics, a few major industry trends seem to have influenced the way users configure headless systems in the home entertainment space today.

1) The ongoing explosion of mobile Apps, on Android, iPhone, Windows phone etc. – To such an extent that “there is an App for anything and everything”! This seems to have turned the tables on the earlier “browser” trend. Who cares for the browser now, there’s an App that does it anyway! And everyone knows it’s a song to just download/install a (whatever) App from the (which ever) Store.

2) The proliferation of home routers – With exploding popularity and plummeting costs, that the term “IP address” today is considered the tech-ABC of modern life. (In the 70’s the term “IP address” used to be high-tech parlance, but today it forms a part of the general vocabulary of a high school kid!) Since IP networking involves a highly standardized set of settings, like DHCP/Static IP, DNS settings etc., a home user can usually manage to set it up without much trouble. Or can at least leave the defaults alone knowing it is good enough to work correctly!

3) Wi-Fi gaining the status of defacto standard in consumer electronics – Thanks mainly to its cable-free, high bandwidth and moderately high range qualities, all ideal for the home segment. Given the massive penetration of Wi-Fi in homes today, networking for most consumer-electronics manufacturers defaults to Wi-Fi, although occasional segments of wired Ethernet could still be present in the home.

Streaming media protocols like Digital Living Network Alliance (DLNA) that work over self-discovery based networking protocols like Universal Plug-n-Play (UPnP) therefore work seamlessly over Wi-Fi, making it a natural choice for most device manufacturers, ranging from the traditional players like Sony, LG, Samsung etc. to recent innovators in this space like Google, Microsoft, Amazon etc. (*) These devices are far from standardized in their function, ranging from simple audio streaming systems, or gaming consoles to full-fledged home theatre systems. Some of these products come with Wi-Fi remotes and configuration (mobile) Apps. The important commonality is that these devices are all accessible on the Wi-Fi network, all the time.

So why configuration for a consumer electronics device! Imagine a user picks up the product from a store and brings it home, so how does the device know the user’s home network details so that it can connect? How could the user tell the consumer electronics device, “this is my network SSID, and this is the passkey”? – A mandatory and generic configuration that is valid for any make/model, for these are the vital pieces of information without which the device cannot join the home network successfully.

Further configuration needs will depend on the nature and functionality of the device. For instance, for a DLNA compliant device, what is the role of that device, whether is it a controller, or a renderer etc. Being non-standard and product-specific configuration settings, a browser based configuration process using menu-driven web-pages may not be intuitive to most home users. Hence custom Apps on popular platforms like Android, iOS etc. (available on the respective App stores for users to download and install) have become in vogue.

The user being a layman here (unlike in the case of a network switch where the user is an experience network administrator) the Config. App needs to provide a zero-learning, intuitive and wizard-driven experience. The Config. App needs to anticipate the user’s concerns, learning curve and proactively push help, like context-sensitive FAQs or “How to” videos, rather than let issues happen and then provide troubleshooting tips. As someone rightly said, “think not how you as an engineer would use it, but how your mom and dad would, and get it to work correctly too”!

The duration of time when the user is configuring the device is called configuration (or config) mode, whereas most of the time while in normal operation serving the home entertainment function, it is said to be in normal mode. To support the web server design pattern for configuration purposes, the device could be switched to become a Wi-Fi Access Point while in config mode. This means that the Config. App (usually a mobile App) can now connect to the device, and the user can set the configurations. Whereas for performing its normal home entertainment function, the device could be switched back to act as Wi-Fi Station, that normally connects to the home router/Wi-Fi Access Point.

Modern Wi-Fi chipsets (a la Broadcom) come with multiple mode operation – Wi-Fi Station (STN) and Wi-Fi Access-Point (AP) modes. Alternatively, a Soft-AP implementation on a single mode chipset would also serve the same purpose, as long as it can be made to switch between modes. The actual switching to config mode could be triggered on a special combo keypress (often with predefined timing) on the device itself. For instance, the Config. App of a consumer electronics device could display the following instruction to the user, “To enter Config mode, press and hold the FFWD and REV buttons simultaneously for more than 10 seconds. The Yellow LED will start blinking to indicate it has entered Config mode “.

From a connectivity view-point it is interesting to note that while in Config mode, the device loses its connection with the home network, since it is now acting as an Access-Point. With the mode switch it has started its own network for configuration Apps to connect, during which its normal functionality like rendering music is not available. It is sometimes even confusing to users, especially if internet access is involved, like Internet Radio etc.

Switching back to normal mode typically happens at reboot. Alternatively it could be implemented as the response to a reboot command while in Config. Mode. Following this mode switch the device regains its ability to play music etc. Another interesting behavior is that when this mode switch/reboot happens, the Smartphone loses its Access Point effectively so it automatically rejoins the home network to which it was connected earlier, a subtle yet standard behavior across Smartphones.

Since the embedded device and its Config. App have now become two peer Wi-Fi stations on the home network, it necessitates a mechanism for the App to know if (and when) the device rejoins the network after reboot. This could be implemented in the device using a broadcast mechanism of datagram messages, which the App could be programmed to detect with a suitable timeout.

Given the need for the Config. App to be available to users on multiple Smartphone platforms like Android, iOS etc. embedded system designers have adopted HTTP-based RESTful APIs as the programming interface to the embedded device. The embedded web-server is in fact a REST Server exporting REST APIs, which are implemented either as GET or as POST requests. The mobile Apps could call these REST APIs, with the data being exchanged as JSON payload of these requests. The embedded firmware that includes the REST Server is usually validated early on in the development cycle using generic REST API Clients (a la Chrome Advanced REST API Client), following which the Android and iOS Apps could be developed and hosted in the respective Stores.

Although not related to configuration, another generic scenario encountered with consumer electronics devices is firmware upgrade, or OTA (Over-the-air) upgrade as it has come to be known. The App typically alerts the user regarding the availability of the latest firmware, based on which the user decides to initiate the upgrade. The firmware image gets downloaded from a designated server of the manufacturer, typically an FTP server, file integrity check is performed after download, the firmware image is written into Flash, the boot loader parameters are modified and the device reboots (often more than once). All of this happens sequentially, silently under the hood, the App finally informing the user regarding the status of the upgrade.

The OTA feature also helps the manufacturer ship the hardware to hit the stores early, while allowing them to conveniently roll out updated features to users. So the next time we buy a device, like a fitness band, and install its App and it triggers off an upgrade (the first-time upgrade often being unsolicited!), we know what’s been happening! The App therefore forms a critical link in the whole upgrade process since the user needs to be kept in the loop always.

Configuration of headless systems has come a long way, the connectivity and Apps migrating across multiple technologies and platforms. In consumer electronics an increasing list of non-Configuration features are getting included in the Apps, such as firmware upgrades, library settings, playlists, favorites etc., making today’s consumer electronics products ever more user friendly and competitive.

(*) – While products from most home audio leaders like Bose, Denon, Yamaha etc. do support the DLNA standard, some of them are known to be moving towards proprietary APIs for media streaming and control. However they will most probably continue to leverage the underlying UPnP layer for device discovery, description and connection services.

Leveraging Kaa IOT platform for quick prototyping

With the explosion of the number of smart devices, Remote diagnostic and Monitoring tools are in great demand, to reduce customer support costs and to improve predictive maintenance aspects. Since the complexity and features of the software/firmware of these smart devices have been growing exponentially, the troubleshooting of the devices is becoming more and more complex. But very few of them have matured sufficiently to handle smart devices. Some of the popular commercial remote troubleshooting solutions are LogMeIn and TeamViewer.

Recently at Sequoia, I was trying to develop a quick POC for remote diagnostic and monitoring solutions for multiple smart devices. The system envisaged would have a diagnostic app running on the smart device, a technician console for communicating with the app and a communication link between them. The challenge I had was twofold (a) the company was a startup and had limited resources (b) the short time line available to develop the POC. After considering several possible solutions, I chose Kaa, an IOT platform, that works well for the quick development of a remote diagnostic and monitoring system.

The main considerations in selecting an IOT platform for a low cost diagnostic system were the following,

  • Multi-platform support.
  • Ease of deployment and development.
  • Integration with 3rd party tools; for example, integration with data pipelines for big data processing and analytics.
  • Scope for feature addition to the existing platform.
  • Actively maintained open source project.
  • Detailed documentation and good community support.

Kaa was the perfect choice with the above attributes. Kaa (http://www.kaaproject.org/) is a 100% open source Apache licensed platform. It has good community support and documentation. Kaa provides integration with other data pipelines like Kafka (http://kafka.apache.org/), Flume (https://flume.apache.org/) and Spark(http://spark.apache.org/). The integration with these data pipelines are very useful, especially when planning an analysis on the collected data. It also integrates well with databases like Cassandra, Mongo DB etc.

There are many features in Kaa like Events, Data Collection, Notifications, Profilingetc. The Events feature of Kaa can be used for transferring bidirectional messages between the technician console and the remote diagnostic app running on the smart device. Both unicast and multicast messages between endpoints are supported through Kaa platform. In Remote Diagnostic System, an end point can be an app (diagnostic) on a smart device; the technician console can be a web dashboard, another app on a smart device or even a desktop application.

Endpoint device <=> Kaa Server <=> Endpoint device/Dashboard/a stand-alone desktop application / Bigdata Server / Database

Since Kaa platform supports multitenancy, only the end point application needs to be developed. All the rest of the communication is taken care of by the Kaa platform.

Troubleshooting information was very easy to obtain with the Kaa platform. This information can be used to identify the technical faults of a particular device or to analyze its maximum life span/stability.    Further the analysis possibilities are unlimited; the frequently occurring issues and the pattern of their solutions can used to  train a machine learning model. Such trained models can be used in the future for automating troubleshooting using a bot.

I was impressed by the robustness of the platform including 3rd party support and easy customization options.

Link to artilce on Linkedin

Business Trends in IoT Ecosystem

Key Trends

There is a trend in the IoT landscape of vendor consolidation around a few major platform vendors. The vendors from different links of the chain collaborate by providing a seamless experience to build, deploy and operate IoT solutions. Many of the major IoT platform leaders have defined end to end system specifications with their ecosystem partners for a secure, scalable and interoperable deployments. This has led to transforming the vertically fragmented and inadequate proprietary markets into well-organized and horizontally scalable open markets. It will enable key players to gain a major share in the IoT market, which is predicted by IDC to reach an annual revenue of USD 1.7 trillion by 2020.

Collaboration, Acquisitions and Partnerships

The acquisition of Wind River and McAfee by Intel, Nest by Google, Solair by Microsoft, Jasper by Cisco and Atmel by Microchip can be seen as strengthening their foothold and consolidation of services in the IoT ecosystem.  These market leaders have also formed alliances and partnerships with other horizontal leaders, SMEs and start-ups to build their own ecosystem. Some of these partnerships are Samsung-SAP, Cisco-GE, Apple-IBM. All these happened because of the awareness of the industry that there was too much fragmentation with companies working in silos. These heavy-duty collaborations are essential for the IoT market to succeed. It is interesting to note that innovation is a key element and the start-ups are going to play a crucial role in that. 2lemetry, Estimote, Digital Lumens, Litbit, Skycatch are some of the leading start-ups getting attention with their innovative ideas.

What the market looks for

When businesses plan for the IoT transformation, the major considerations are the ease of interoperability, seamless integration, maintainability and scalability factors. The other factors being reducing risk and lowering the cost of implementation and operation.

The market needs vendors, device manufacturers and services providers who can help to quickly rollout the IoT solution with the essentials. These include automated discovery and provisioning of sensors and actuators, device control, seamless storage and processing of data, security of data and devices, analytics capability and monetizing of the services. This is where these collaborations are going to benefit the end customer and the industry as a whole.

The Benefit and Value Add

The crux of this collaboration is the desire of every player in IoT, big or small, for a transformation from a product business into services business, seeing the long term and continuous monetization opportunity. In essence, the alliances of platform leaders with vendors at different planes of the spectrum will influence the choices of picking devices, software, services and operators in an end to end IoT solution. The positive note about all these collaborations and alliances is that it is going to eventually help the end users with competitive pricing, faster deployments, innovative and effective solutions.

Enabling Big Data in your IOT Strategy

IOT and Big data are considered as two sides of a same coin. For some industry requirements, you won’t be able to differentiate whether it is IOT or big data solution. At the same time, in some cases your IOT solutions may not be having any big data use cases. In such cases, it is always a tough call for the IOT strategist to decide on the feasibility of considering big data in their system.

I suggest to keep your data big data friendly.

What does big data friendly mean?

Current big data solutions are spending lot of efforts to clean up their large amount of complex & messy data. It happened just because the data was not collected big data friendly.  We should learn from the mistakes of others!!! Your data may not be “big” now, but it is growing  and soon  will be large  for any big data processing in future. Industry already started adopting new approaches, moving from descriptive to predictive and prescriptive analytics. We should prepare our data for various big data analytics in future to improve our business value.

How can we make our IOT strategies big data friendly?

There are a few design considerations,

1. Finding ‘target-rich’ data :-

All data may not be worth saving for future. Organizations need to focus on the high-value ‘target-rich’ data. Business analyst and data scientist should work on identifying the high value data which can be an asset for the organization in near future.

2. Data saving strategies :-

How data is stored is a major decision point for the big data system.

2.1 Data Format

Saving text data in formats like JSON data won’t be easy to handle when the data become huge. We should also consider using more big data friendly formats like Parquet, Avro, ORC etc

2.2 Dynamic schema for version handling

The data veracity increases when the system grows with different versions have different fields added or renamed or removed. Manually handling schema is always an overhead. Self describing data formats will be a suitable solution. We should consider using data formats having dynamic schema generation capability (like parquet).

2.3 Partitioning strategy of data

Data partitioning strategy should be defined to having better segmentation of data for easy processing.

3. Security considerations.

Data security is one of the major concerns of both big data and IOT systems. We should implement the privacy by design. Privacy impact assessment and data anonymization are few things to be considered.

In a multi tenant environment, you don’t know how fast your data is growing and become matured as big data. Preparing your data for future will enable to extract maximum value out of it.

“Information is wealth”, Sequoia Applied Technologies will help design solutions to Capture it before you lose it!! .

Link to Article on Linkedin