Overview
  • Updated on 24 Jul 2019
  • 7 minutes to read
  • Contributors
  • Print
  • Share
  • Dark
    Light

Overview

  • Print
  • Share
  • Dark
    Light

Overview

Today’s applications are required to be highly responsive and always online. To achieve low latency and high availability, instances of these applications need to be deployed in datacenters that are close to their users. These applications are typically deployed in multiple datacenters and are called globally distributed.

Globally distributed applications need a geo distributed fast data platform that can transparently replicate the data anywhere in the world to enable the applications to operate on a copy of the data that's close to its users. Similarly the applications need geo-replicated and local streams to handle pub-sub, ETL and real-time updates from the fast data platform.

C8 is a fully managed geo-distributed fast data service with turnkey global distribution and transparent multi-master replication. You can run globally distributed, low-latency workloads within C8. This article is an introduction to using C8 with pyC8 (python driver).

pyc8_tutorial

In pyC8, a document is a Python dictionary that is JSON serializable with the following properties:

  • Contains the _key field, which identifies the document uniquely within a specific collection.
  • Contains the _id field (also called the handle), which identifies the document uniquely across all collections within a fabric. This ID is a combination of the collection name and the document key using the format {collection}/{key} (see example below).
  • Contains the _rev field. C8 supports MVCC (Multiple Version Concurrency Control) and is capable of storing each document in multiple revisions. Latest revision of a document is indicated by this field. The field is populated by C8 and is not required as input unless you want to validate a document against its current revision.

Here is an example of a valid document:

    {
        '_id': 'students/bruce',
        '_key': 'bruce',
        '_rev': '_Wm3dzEi--_',
        'first_name': 'Bruce',
        'last_name': 'Wayne',
        'address': {
            'street' : '1007 Mountain Dr.',
            'city': 'Gotham',
            'state': 'NJ'
        },
        'is_rich': True,
        'friends': ['robin', 'gordon']
    }

Edge documents (edges) are similar to standard documents but with two additional required fields _from and _to. Values of these fields must be the handles of "from" and "to" vertex documents linked by the edge document in question (see :doc:graph for details). Here is an example of a valid edge document:

    {
        '_id': 'friends/001',
        '_key': '001',
        '_rev': '_Wm3dyle--_',
        '_from': 'students/john',
        '_to': 'students/jane',
        'closeness': 9.5
    }

Pre-requisite

Let's assume your tenant name is demotenant and root user password is demopwd.

Download Python Client

pyC8 requires Python 3.5+, and Python 3.6 or higher is recommended

To install pyC8, simply run

 $ pip3 install pyC8

or, if you prefer to use conda:

conda install -c conda-forge pyC8

or pipenv:

pipenv install --pre pyC8

Once the installation process is finished, you can begin developing C8 fast data applications in Python.

Connect to C8

The first step in using C8 is to establish a connection to a local region. When this code executes, it initializes the server connection to the region URL you sepcified.

from c8 import C8Client

print("Connect to C8...")
client = C8Client(protocol='https', host='MY-C8-URL', port=443)

Create Geo-Fabric

Let's now create a user demouser with password demouserpwd. Then create geo-fabric called demofabric by passing a parameter called dclist. The fabric demofabric is created in all the regions specified in the dclist. You can assign rw permissions over demofabric to demouser.

from c8 import C8Client

print("Connect to C8...")
client = C8Client(protocol='https', host='MY-C8-URL', port=443)

print("Get tenant...")
demotenant = client.tenant(name='demotenant', fabricname='_system', username='root', password='demopwd')

print("Create user...")
if not demotenant.has_user('demouser'):
    demotenant.create_user(username='demouser', password='demouserpwd', active=True)

print("Create geo fabric...")
if not demotenant.has_fabric('demofabric'):
    demotenant.create_fabric(name='demofabric', dclist=demotenant.dclist())

print("Add permissions for user...")
demotenant.update_permission(username='demouser', permission='rw', fabric='demofabric')
Note:
The call to demotenant.dclist() returns a list of all Edge Locations (AKA Datacenters) deployed in the Macrometa Fabric.
Note:
You need to check existence for both demouser i.e., has_user(demouser) and demofabric i.e., has_fabric(demofabric) to ensure we do not create duplicate resources

Get GeoFabric Details

To get details of fabric geo-fabric

from c8 import C8Client
client = C8Client(protocol='https', host='MY-C8-URL', port=443)

print("Get geo fabric...")
fabric = client.fabric(tenant='demotenant', name='demofabric', username='demouser', password='demouserpwd')

print("Get geo fabric details...")
print(fabric.fabrics_detail())

Create Collection

We can now create collection in the fabric. To do this, first you connect to demofabric under demotenant with user as demouser and password demouserpwd. Then create a collection called employees.

The below example shows the steps.

employees = fabric.create_collection('employees') # Create a new collection named "employees".

Create Index

Let's add a hash_index called emails to our collection employees. Please refer to user guide for details on other available index types.

employees.add_hash_index(fields=['email', '_key'], unique=True) # Add a hash index to the collection.

Populate Collection

Let's insert documents to the employees collection as shown below.

employees.insert({'_key':'Jean', 'firstname': 'Jean', 'lastname':'Picard', 'email':'jean.picard@macrometa.io'})
employees.insert({'_key':'James', 'firstname': 'James', 'lastname':'Kirk', 'email':'james.kirk@macrometa.io'})
employees.insert({'_key': 'Han', 'firstname': 'Han', 'lastname':'Solo', 'email':'han.solo@macrometa.io'})
employees.insert({'_key': 'Bruce', 'firstname': 'Bruce', 'lastname':'Wayne', 'email':'bruce.wayne@macrometa.io'})

Retrieve documents using C8QL

C8QL is C8's query language. You can execute C8QL query on our newly created collection employees to get its contents.

The query FOR employee IN employees RETURN employee is equivalent to SQL's SELECT query.

from c8 import C8Client

print("Create connection...")
client = C8Client(protocol='https', host='MY-C8-URL', port=443)

print("Get geo fabric...")
fabric = client.fabric(tenant='demotenant', name='demofabric', username='demouser', password='demouserpwd')

print("Execute C8QL query...")
cursor = fabric.c8ql.execute('FOR employee IN employees RETURN employee') 

print("Retrieve documents...")
docs = [document for document in cursor]
print(docs)

Real-time Database

Example for real-time updates from a collection in fabric:

from c8 import C8Client
import warnings
warnings.filterwarnings("ignore")

def callback_fn(event):
    print(event)

print("Create connection...")
client = C8Client(protocol='https', host='MY-C8-URL', port=443)

print("Get geo fabric...")
fabric = client.fabric(tenant="demotenant", name="demofabric", username="demouser", password='demouserpwd')

print("Subscribe to receive updates on employees collection...")
fabric.on_change("employees", callback=callback_fn)

Create Global & Local Streams

Creating streams in C8 is a 1 step operation. The stream can be either a local stream or could be a geo-replicated stream.

from c8 import C8Client

demo_stream = 'demostream'  #Name of the Stream
print("Create connection...")
client = C8Client(protocol='https', host='MY-C8-URL', port=443)

print("Get geo fabric...")
fabric = client.fabric(tenant='demotenant', name='demofabric', username='demouser', password='demouserpwd')

print("Create geo-replicated & local streams in demofabric...")
fabric.create_stream(demo_stream, local=False)
fabric.create_stream(demo_stream, local=True)

print("Get streams...")
streams = fabric.streams()
print("streams:", streams)

Publish Messages

Example to publish documents to a stream. The stream can be either a local stream or could be a geo-replicated stream.

from c8 import C8Client
import time
import warnings
warnings.filterwarnings("ignore")

print("Create connection...")
client = C8Client(protocol='https', host=MY-C8-URL, port=443)

print("Get geo fabric...")
fabric = client.fabric(tenant="demotenant", name="demofabric", username="demouser", password='demouserpwd')

print("Create producer...")
stream = fabric.stream()
producer = stream.create_producer("demostream", local=True)

print("Publish messages to stream...")
for i in range(10):
    msg = "Hello from  user-->" +  "("+ str(i) +")"
    producer.send(msg.encode('utf-8'))
    time.sleep(10) #sec

Subscribe to Stream

Example to subscribe documents from a stream. The stream can be either a local stream or could be a geo-replicated stream.

from c8 import C8Client
import time
import warnings
warnings.filterwarnings("ignore")

print("Create connection...")
client = C8Client(protocol='https', host=MY-C8-URL, port=443)

print("Get geo fabric...")
fabric = client.fabric(tenant="demotenant", name="demofabric", username="demouser", password='demouserpwd')

print("Get stream ...")
stream_collection = fabric.stream()

print("Subscribe to stream ...")
subscriber = stream_collection.subscribe("demostream",local=True, subscription_name="demosub", consumer_type= stream_collection.CONSUMER_TYPES.EXCLUSIVE)
#you can subscribe using consumer_types option.

print("Print received documents ...")
for i in range(10):
    msg = subscriber.receive()
    print("Received message '{}' id='{}'".format(msg.data(), msg.message_id()))
    subscriber.acknowledge(msg)

Spot Collections

Create a geo-fabric with spot region capabilities. Then create a collection that is designated as a spot collection. A geo-fabric can contain both regularand spot collections.

from c8 import C8Client

# Initialize the client for C8DB.
client = C8Client(protocol='https', host='MY-C8-URL', port=443)

#Create a geo-fabric and pass one of the spot regions. You can use the SPOT_CREATION_TYPES for the same. If you use AUTOMATIC, a random spot region will be assigned by the system.

# If you specify None, a geo-fabric is created without the spot properties. If you specify spot region,pass the corresponding spot region in the spot_dc parameter.

print("Get tenant...")
sys_tenant = client.tenant(name='demotenant', fabricname='demofabric', username='demouser', password='demouserpwd')

print("Create Geo-Fabric with SPOT capability...")
fabric = client.fabric(tenant='demotenant', name='demofabric',  username='demouser', password='demouserpwd')
local_region = sys_tenant.dclist_local()
fabric.create_fabric('spot-geo-fabric',  dclist=sys_tenant.dclist(), spot_creation_type= fabric.SPOT_CREATION_TYPES.SPOT_REGION, spot_dc=local_region["tags"]["url"].split(".")[0], users = [{"username": "demouser", "password": "demouserpwd", "active": True}])

print("Create spot collection ...")
spot_collection = fabric.create_collection('spot-collection', spot_collection=True)

print("Insert documents into spot collection ...")
spot_collection.insert({'firstname': 'Jean', 'lastname':'Picard', 'email':'jean.picard@macrometa.io'})
spot_collection.insert({'firstname': 'James', 'lastname':'Kirk', 'email':'james.kirk@macrometa.io'})
spot_collection.insert({'firstname': 'Han', 'lastname':'Solo', 'email':'han.solo@macrometa.io'})
spot_collection.insert({'firstname': 'Bruce', 'lastname':'Wayne', 'email':'bruce.wayne@macrometa.io'})

print("Execute C8QL query...")
cursor = fabric.c8ql.execute('FOR employee IN spot_collection RETURN employee') 

print("Retrieve documents...")
docs = [document for document in cursor]
print(docs)

Interactive Program

You can use below to play with the pyC8 driver. I would recommend to get your own developer account on Macrometa free tier though instead of using the guest account. The data in guest account will be purged daily.

Was this article helpful?