Multimedia Mobile application using Low Power Bluetooth(BLE)

… in a museum. You walk by an painting suddenly your phone becomes the voice of the artist and begins to speak to you about the piece…. Bringing art to the guest.

In 2008 I developed an application that used RFID to trigger events on a mobile device(PDA).  The main purpose was to be an Electronic Docent, a museum guide. Exhibit information delivered directly to the guest.

Unfortunately RFID never became a consumer  friendly technology. Fast forward to 2016, smart phones are prevalent and low power bluetooth(ble) devices are becoming ever more popular. In January myself and  two others began development on a new version of the application.

The PDA has been replace by smart phones and tablets. Both iOS and Android hold major positions in this area. Both have support for standard Bluetooth as well as BLE.

How it works

The application running on the device is designed to look for BLE tags.  When one is located a request is made to a server to search the database for the tag id. If the id is located information about the media is returned and the user can select if they want to view the media or not.

how-it-works1

The tags and media have to be associated.This is done by personnel managing the location. They understand both the content and how they would like it displayed to the visitor.

 

how-it-works2

 

One of the biggest decisions was how to develop the mobile portion of the application.

  1. Native, iOS and Android
  2. Cross platform framework: Xarmin,QT
  3. Javascript/HMTL5 framework: Apache Cordova ( formerly  Phonegap), Ionic.

Native

Until a few years ago mobile applications were required to be developed in Java  or Obj C. Apple refused to accept applications cross compiled or interpreted into Obj C. The draw back is that an application had to be developed twice. Maintenance was much harder since it required twice the effort in coding and QA.

On the other side,native applications had the ability to interact with the devices hardware, sound, touch, gps, and accelerometer.

Cross platform framework

Frameworks such as Xarmin and QT give the developer to write one application and deploy it to multiple mobile platforms.

Xarmin: Based around C# and created by team that created Mono. Xarmin takes C# code and creates native code for iOS or Android. Microsoft now owns Xarmin and has integrated it into its Visual Studio IDE.

QT: This has long been a popular framework for developing applications for Windows, OSX and Linux. When mobile support was added there became license issues. Also QT has less of a native look and feel.

Javascript/HMTL5 framework: Tools such as Ionic use the Angular.js framework and Cordova libraries to create cross platform applications. The key to the success has been the Cordova(Phonegap) libraries. These provide access to the device hardware which lets the application behave more like native code.

We chose Ionic. There were too many issues with either Xarmin or QT. Developing two separate native applications was out of the question.

Serving up data

Once the mobile application find a BLE tag it needs to get the information associated with the tag. This means an application server. This was a simple choice, Java, Hibernate ,MySQL and Tomcat. This combination is proven, solid and will work well on something like AWS. One advantage to MySQL is that AWS’s Aruora database is MySQL  compatible and an easy replacement if very high performance is required.

Server side

Using Java and Hibernate makes the server work pretty straight forward. The code is built on layers. Entities,DAO, Service, Controller.

Entity

Each entity represents a table in the database.

  • ExhibitTag
    •   This represents a single ble tag
  • ExhibitTagMedia
    • This represents the media associated with tag. A tag could have more than one media component.
  • Location
    • This represents the location of the tag
  • Organization
    •  This  represents the site or organization managing the tags.
package com.tundra.entity;
import java.io.Serializable;
import java.util.Date;
import java.util.Set;
import javax.persistence.Basic;
import javax.persistence.CascadeType;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.FetchType;
import javax.persistence.Id;
import javax.persistence.JoinColumn;
import javax.persistence.ManyToOne;
import javax.persistence.OneToMany;
import javax.persistence.Table;
import javax.persistence.Temporal;
import javax.persistence.TemporalType;
import com.fasterxml.jackson.annotation.JsonIgnore;

@Entity
@Table(name = "exibittag")
public class ExhibitTag implements Serializable {
private static final long serialVersionUID = 1L;
 @Id
 @Basic(optional = false)
 @Column(name = "Id")
 private Integer id;
 @Basic(optional = false)
 @Column(name = "Name")
 private String name;
 @Basic(optional = false)
 @Column(name = "Tag")
 private String tag;
 @Basic(optional = false)
 @Column(name = "Description")
 private String description;
 @Basic(optional = false)
 @Column(name = "Created")
 @Temporal(TemporalType.TIMESTAMP)
 private Date created;
 @Basic(optional = false)
 @Column(name = "Updated")
 @Temporal(TemporalType.TIMESTAMP)
 private Date updated;
 @JsonIgnore
 @JoinColumn(name = "Location_Id", referencedColumnName = "Id")
 @ManyToOne(optional = false, fetch = FetchType.EAGER)
 private Location location;
 
 @OneToMany(cascade = CascadeType.ALL, mappedBy = "exhibitTag", fetch = FetchType.EAGER)
 private Set<ExhibitTagMedia> exhibitTagMediaSet;

//
// setters and getters removed

@Override
 public int hashCode() {
 int hash = 0;
 hash += (id != null ? id.hashCode() : 0);
 return hash;
 }

@Override
 public boolean equals(Object object) {
 if (!(object instanceof ExhibitTag)) {
 return false;
 }
 ExhibitTag other = (ExhibitTag) object;
 if (this.id == null && other.id == null) {
 return super.equals(other);
 }
 if ((this.id == null && other.id != null) || (this.id != null && !this.id.equals(other.id))) {
 return false;
 }
 return true;
 }

@Override
 public String toString() {
 return "Exibittag[ id=" + id + " ]";
 }
}

 

DAO

Spring will create the query for FindByTag() automatically

package com.tundra.dao;
import java.util.List;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.transaction.annotation.Transactional;
import com.tundra.entity.ExhibitTag;

@Transactional("tundraTransactionManager")
public interface ExhibitTagDAO extends JpaRepository<ExhibitTag, Integer> {
 List<ExhibitTag> findByTag(String tag); 
}

Service

The service layer is how the controller will interface with the server.

package com.tundra.service;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.List;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import com.tundra.dao.ExhibitTagDAO;
import com.tundra.dao.ExhibitTagMediaDAO;
import com.tundra.dao.OrganizationDAO;
import com.tundra.entity.ExhibitTag;
import com.tundra.entity.ExhibitTagMedia;
import com.tundra.entity.Organization;
import com.tundra.response.ExhibitTagSummaryResponse;

@Service
public class TundraServiceImpl implements TundraService {
 
 @Autowired
 ExhibitTagDAO exhibitTagDAO;
 @Autowired
 private OrganizationDAO organizationDAO;
 @Autowired
 ExhibitTagMediaDAO exhibitTagMediaDAO; 
 /* (non-Javadoc)
 * @see com.tundra.service.TundraService#findAllOrganizations()
 */
 @Override
 public List<Organization> findAllOrganizations() {
 return organizationDAO.findAll();
 }
 
 /* (non-Javadoc)
 * @see com.tundra.service.TundraService#findOrganization(int)
 */
 @Override
 public Organization findOrganization(int id) {
 return organizationDAO.findOne(id);
 }
 @Override
 public List<Organization> findByName(String name) {
 return organizationDAO.findByName(name);
 }
 @Override
 public List<Organization> findByNameAndCity(String name, String city) {
 return organizationDAO.findByNameAndCity(name, city);
 }
 @Override
 public ExhibitTag findByTag(String tag) {
 ExhibitTag et = null;
 List<ExhibitTag> list = exhibitTagDAO.findByTag(tag);
 if( list != null && list.size() ==1){
 et = list.get(0);
 }
 return et;
 }

@Override
 public List<ExhibitTag> findAllTags() {
 return exhibitTagDAO.findAll();
 }
@Override
 public ExhibitTagMedia findMediaByTag(String tag) {
 ExhibitTagMedia media = null;
 List<ExhibitTagMedia> list = exhibitTagMediaDAO.findByExhibitTag(tag);
 if( list != null && list.size() ==1){
 media = list.get(0);
 }
 return media;
 }
@Override
 public ExhibitTagSummaryResponse findSummaryByExhibitTag(String tag) {
 ExhibitTagSummaryResponse summary = null;
 List<ExhibitTagSummaryResponse> list = exhibitTagMediaDAO.findSummaryByExhibitTag(tag);
 if( list != null && list.size() ==1){
 summary = list.get(0);
 }
 return summary;
 }
}

Controller

The controller layer  represents the REST layer. The mobile app will interface with the server via the controller.

package com.tundra.controller;
import java.io.Serializable;
import java.util.List;
import javax.servlet.http.HttpServletResponse;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.ResponseBody;
import com.tundra.entity.ExhibitTag;
import com.tundra.entity.ExhibitTagMedia;
import com.tundra.response.ExhibitTagSummaryResponse;
import com.tundra.service.TundraService;

@Controller
@RequestMapping("/tag")
public class ExhibitController implements Serializable {
private static final String ERROR_PREFIX = "Whoops : ";
 
 private static final long serialVersionUID = 1L;
 @Autowired
 private TundraService tundraService;
 
 @RequestMapping(value="/{tag}", method=RequestMethod.GET)
 public @ResponseBody ResponseEntity<?> getExhibitTagByTagId(HttpServletResponse httpResponse, @PathVariable(value="tag") String tag) {
 try {
 return new ResponseEntity<ExhibitTagSummaryResponse>(tundraService.findSummaryByExhibitTag(tag),HttpStatus.OK);
 } catch (Throwable t) {
 return new ResponseEntity<String>(ERROR_PREFIX + t.getMessage() ,HttpStatus.INTERNAL_SERVER_ERROR);
 }
 }
 @RequestMapping(value="/media/{tag}", method=RequestMethod.GET)
 public @ResponseBody ResponseEntity<?> getExhibitMediaByTagId(HttpServletResponse httpResponse, @PathVariable(value="tag") String tag) {
 try {
 return new ResponseEntity<ExhibitTagMedia>(tundraService.findMediaByTag(tag),HttpStatus.OK);
 } catch (Throwable t) {
 return new ResponseEntity<String>(ERROR_PREFIX + t.getMessage() ,HttpStatus.INTERNAL_SERVER_ERROR);
 }
 }
 @RequestMapping(value="/list", method=RequestMethod.GET)
 public @ResponseBody ResponseEntity<?> getExhibits(HttpServletResponse httpResponse) {
 try {
 return new ResponseEntity<List<ExhibitTag>>(tundraService.findAllTags(),HttpStatus.OK);
 } catch (Throwable t) {
 return new ResponseEntity<String>(ERROR_PREFIX + t.getMessage() ,HttpStatus.INTERNAL_SERVER_ERROR);
 }
 }
}

 

With the server code in place its time to look at the mobile app.

As started earlier we are using the Ionic framework. It is based on Javascript/Angular.

The structure of a Ionic project is shown below.

 

The areas we change are app.js, controller.js and service.js. Index.html is modified only slightly to include our files.

<!-- cordova script (this will be a 404 during development)-->
 http://cordova.js
 http://lib/angular-base64/angular-base64.js

<!-- your app's js -->
 http://js/app.js
 http://js/controllers.js
 http://js/services.js

The templates folder holds the html files for the various screens. Since we started with a tabbed ionic project we have two core html templates, tab.html and tab-dash.html.The tab format allows tabbed pages as a navigation. We are not using this format and  will be renamed later on.

tab.html

<ion-tab title=”My Docent” icon-off=”ion-ios-pulse” icon-on=”ion-ios-pulse-strong” href=”#/tab/dash”>
<ion-nav-view name=”tab-dash”></ion-nav-view>
</ion-tab>

The main screen is in tab-dash.html

 <ion-content>
 <ion-header-bar align-title="center" class="bar-stable">
 
Search
<h1 class="title">Available Exhibits</h1>
</ion-header-bar> </ion-content>

The screen is very basic.

tundra-ui

The other screens are used to represent the media types, text,video,and audio.html. Here is an example of a text view.

tundra-ui-2

The app.js file is loaded first and sets up the basic structure. The application uses the Ionic Bluetooth Low Energy (BLE) Central plugin for Apache Cordova . If the app is running on a real mobile device(not in a browser on a PC) the object ‘ble’ will be defined. On a PC this will not be valid. The app.js  run function will check for this.

 if(typeof(ble) != "undefined"){
     ble.isEnabled(
      function () {
        document.getElementById("bleStatus").style= "color:green;";
     },function () {
         document.getElementById("bleStatus").style= "color:red;";
    }
   );
 }

 

The controller layer manages the controls from the html(UI) code.

Example: It the main html file there is a button to start scanning.
<button ng-click=”startScanning()” class=”button”>Search</button>

In the controller there is the startScanning function. The BLEService is located in the service layer.

$scope.startScanning = function () {
    BLEService.connect(function(exibitTags) {
    $scope.exibitTags = exibitTags;
    console.log(exibitTags);
 });
 
   $scope.myText = "startScanning";
    console.log($scope.myText);
    isScanning = true; 
 };

In the service layer.

.service("BLEService", function($http){
  if (typeof(ble) != "undefined") {
     ble.scan([], 30, onConnect, onError);

The onConnect function returns a list of Bluetooth tags  located.

Once the list of devices is returned, the REST service is called to check the tags against the database. The server returns:

  • Organization Name
  • Location Name
  • Exhibit TagName
  • Exhibit TagId
  • Exhibit Tag
  • Exhibit Tag MimeType

The user selects which exhibit they want to view.

Testing the app locally

Ionic can run the app locally by using the command ‘ionic serve’ from the project folder.

C:\Users\rickerg0\workspace\Tundra>ionic serve
******************************************************
The port 35729 was taken on the host localhost - using port 35730 instead
Running live reload server: http://localhost:35730
Watching: www/**/*, !www/lib/**/*, !www/**/*.map
√ Running dev server: http://localhost:8100
Ionic server commands, enter:
 restart or r to restart the client app from the root
 goto or g and a url to have the app navigate to the given url
 consolelogs or c to enable/disable console log output
 serverlogs or s to enable/disable server log output
 quit or q to shutdown the server and exit

The basic screen as viewed in FireFox.

tundra-ui

Deploy the app to an Android device from Windows

Make sure the device is connected via the USB port. Also set the developer option on the device. If you don’t do this last step the device will not allow Ionic to connect.  From the terminal issue the command ‘ionic run android’. This will build the apk file and install it on the device.

 

 

About gricker

Living and learning
This entry was posted in Uncategorized. Bookmark the permalink.