Categories
AI Uncategorized

Let AI come up with your next domain name!

Intro

Before I started on my own AI-related venture, I didn’t know anything about it. I knew what it was, but I couldn’t begin to explain how things worked. Turns out, I can learn a lot when I can add “motivation” to the list of ingredients (no pun intended). Everything that interests me, will stick in my mind forever.

One of the most beautiful things was that I’m doing exactly the same thing as when I was that naive 13-year old kid that just started programming. I learned bits and pieces about programming languages, if-else clauses, loops etc, until the pieces fell together. Then I started building stuff, I didn’t look up or around: this was the destiny for me.

Now I’m 25 and I’m doing the same thing with AI, and it’s awesome. I gave my first AI-related talk 2 weeks ago.


One of the things I wanted to do was something that hasn’t been done before (at least not publicly).

So I built an AI that comes up with brandable domain names. Which from now on I refer to as NameAI

When I started “NameAI”. I didn’t expect much, but I was really suprised by the results! After half an hour I already got a lot of good names (like kixely.com).

If you want to dive straight in the source-code, see the bottom of this post with the Github link!

The research

I started with a fairly common stack for AI-research: Python, Keras (with Theano backend), and Jupyter. You don’t need a strong GPU, a CPU is enough (everything >= Core i5 would be quick).

My dataset is a text-file consisting of ~25k brandable domain names, with 1 domain per line. Unfortunately, I can’t share the dataset publicly, since I ripped it from various websites (like brandbucket.com) but you can easily make your own.

After a brief moment of fiddling around with the parameters and data pipeline, I found the best way to achieve good name prediction. Our model consists of a single GRU layer with 128 units (which is quite small). My goal was that NameAI would learn the linguistics of what makes a name brandable, in which case such a small model is a perfect fit.

The data pipeline consists of our ~25k domain names corpus being split up at each character (i.e codebuffet.co will be split like this: [ ‘c’, ‘o’, ‘d’, ‘e’, ‘b’, ‘u’, ‘f’, ‘f’, ‘e’, ‘t’, ‘.’, ‘c’, ‘o’ ]).

In the initial run, we count the length of the longest domain name in the corpus, and then left-pad every shorter domain with a ‘_’ until it’s as long as the longest domain. This way each domain name is equally long, and we can also use the _ to determine how long our name will (or has to) be.

The data flows through the network in a time-series window of 3 characters.

Time-series window

A window of 3 characters means that the network will keep in mind the past 3 characters, while predicting the next single character. For domain names, I found this one of the best ways, since many of the vowels remain preserved in steps of 3 characters in the English language.

The training itself is unsupervised, which means that the input data is not labeled, and there is no validation. There is no way to measure the accuracy except by looking at the domains that come out of it, and see if it matches your satisfaction.
I did this, because we don’t want our network to validate against our input, we want it to come up with new names, and this is exactly where unsupervised learning is really good at: For example, while it can’t figure out what a face is from a series of pictures (without some validation-set marking the area of the face), it can cluster all the faces from pictures into different groups and output combinations of those groups.
This is the same with our names: imagine ‘.com’ at the end of each name, but also ‘ly’ which is quite common to append to English words, like timely or bitly.

Start training

After each iteration we ask NameAI to make a domain name starting with each letter of the alphabet (including difficult ones like X, Q, Z etc, we really want to push it’s limits here).

After the first iteration we have a lot of gibberish, which is no surprise. It looks like this:

Gibberish on the first iteration.

After about 5 iterations we really start to see some results:

After about 5 iterations, this is your output

Conclusion

This has been an amazing project to do! With a decent amount of examples in a corpus, you can get as many different names as you like. It probably works too for other type of names, like Pokemon names.

Maybe my (or your!) next startup name will come from Name AI 🙂 We’ll see.

I’m glad to share this with you guys, if you have any questions or suggestions for a follow-up – just let me know!

The full source code is at: My Github

Categories
Uncategorized Web Dev

New website and logo showcase: A brand new brand

Yesterday, the newest CodeBuffet website has been put online! The entire website, this blog and logo has been changed. There are a number of nice little technological achievements in the new website, and I’d like to share those with the world!

5 years of CodeBuffet

The new website

The main differences between the new and the old version, is that the new version is completely home-made. The previous version was just a custom WordPress theme.

1 big slide show

The first thing you’ll notice when you go to codebuffet.co, is that you can slide back and forth into different slides while being on the homepage. On some slides, you can scroll down to read more about the specific slide (i.e scrolling down the homepage slide will give you an intro to the company)

I have had this kind of design in my mind since 4 years, but it was only now that I had the feeling that I had implemented in the most practical way. I see this kind of design as quite risky. I wanted to try it nonetheless.

The colors of the page adjusts to that of the content

One of the reasons I wanted a ‘big slideshow’ was to make the portfolio come to life in the best way possible. One of the things that occurred to me when you have such a big space reserved for your portfolio, you basically have 99% content and a 1% frame.

So instead of adjusting the content to fit the page, I decided to adjust the page to fit the content.

2 different slides makes the side adjust it’s color.

There is something very interesting in the way this works: The logo on the bottom-center is not an image, it’s a font. The reason for this is because you can’t instantly change the color of an image in html/css/js without using a canvas or pre-processed (colored) versions of an image (source). Using a canvas with color transitions would be a disaster regarding performance. With a font, I can change and transition the ‘text color’ back and forth to whatever I want, without having performance penalties, even on mobile.

Logo changing colors

Conclusion

Overall, there are some very neat developments being done on this website, it’s by no doubt become a creative outlet for my company’s web development efforts. It’s been up just one day (at this time of writing). I hope you like it, any feedback is welcome 🙂

Categories
Web Dev

I’ve never been so thankful for today’s webdev tools and browsers.

We’ve been seeing countless of rants about JS and about the tons of web dev libraries and modules on npm, especially the horror it causes. I never really agreed with them, but today made me so thankful for today’s web dev economy that if it were a person I would do anything he/she asked for the rest of my life.

I’m currently making a slide show page for a client that is a very large library, that runs custom-made computers and screens from some supplier I forgot their name about.
They have around 6 of those screens in the main lobby.

For them it’s very useful (it has a fully fledged gui where you can tell which screen should show which slides and when)

I was asked to make a custom slide-show page for one of the screens. It should contain information about the visitors who checked in among other stuff.
I started checking out the company that made the displays first (which name I now forgot, was a while ago) and I couldn’t find any information targeted towards developers, no documentation, just a buy button and some features.

So it was clear to me we’re dealing with some kind of very proprietary display. They also didn’t want to move to a more open platform for that particular display, like Raspberry Pi.

I ran a diagnostics page to determine the software it was running.

unnamed

Ouch. I figured it wouldn’t be that bad, I just set up a simple Webpack project with all my favorite preprocessors (Jade, Stylus with Nib, CoffeeScript) and be able to share some code from the other stuff I made for them.

I wish it were that simple… I couldn’t even make rounded corners! I had to download a shim for that (CSS3Pie)
Oh, and the whole script didn’t run unless I installed a polyfill for console.log.

Web fonts? I haven’t even figured that one out yet.

It took me hours to figure this all out, and that including some concessions being made (we removed the partially transparent backgrounds we had in our design)

This really brought me back. I never imagined I had to deal with IE again. I forgot about it. Now I know what it’s like again.

Categories
All Hail the Pioneers Life & Low-Tech Uncategorized

All hail the Pioneers: Day 3

This is the last part of a 3-day series. Click here to view all posts in the series.
About 2 days late, sorry about that 🙂

The last day! Picture made by Hans Stockmans, used with permission.
The last day! Picture made by Hans Stockmans, used with permission.

I woke up and rushed out of the tent quickly, as this was my last day. I really wanted to take it all in this day. Every day there are new workshops that you can do.
These workshops vary from technology, entrepreneurship to philosophy and spirituality.  Today was the first day I actually could do them all day.

I decided to only focus on the low-tech workshops (that didn’t have anything to do with technology or entrepreneurship).

As I walked still half-asleep to the panel with the workshops for today, I couldn’t really pick one. A nice lady behind me shouted “Who’s in for yoga?!”.

“Great!” I thought, now we got something going on!

About an hour later we were back at the main campsite, the yoga in the morning really helped to wake up, I even decided to keep doing it at home as replacement to morning gymnastics because it’s less intense but gives the same result. I think it’s around 10 in the morning now.

I don’t remember if the second workshop was first or the breakfast. Needless to say, the breakfast was very good, as was the lunch. This was the first day I actually could eat breakfast or lunch without being in a hurry. I think there were more than 4 different kinds of bread and around 10 different things you can put on it. So for everyone’s taste there was something.

The second and my last official workshop was a spiritual one, called “Travel to your past life”. Now I would like to mention first that you don’t have to believe in reincarnation. For me, if something brings you the feeling you intended to get out of it, from that point, it doesn’t matter if it’s real or not anymore. I can understand that most of my readers would lose me from this point on 🙂 All I can say, I would lose it too if I would read this without being there myself. So try to focus on the result and the feelings rather than on the believes, if that’s not your thing.

We were with a big group, I think around 20 people isn’t exaggerated.
We gathered around a tent in the main campsite, the instructor, called Anita, introduced us to the workshop. We were explicitly warned that not everyone could travel to their past live because of the large amount of people that joined today, and moved to the same location as where the yoga took place in the morning.

I’m not going trough everything we have done there, as that would make the post too long, so we focus only on the first and a bit on the second person who got a reading before we move to my experience.

After everyone sat in circles around Anita, the first lady who got a reading was chosen. She lied down next to Anita (who was sitting besides her) and trough different routines she was able to read what the unconscious mind was telling you. Every time she tells a century like ‘1500’ and by lifting your arm she could tell if that was the year your past life was in.

After a while, the girl was able to describe where she was and what she was feeling. I noticed that she was more capable to describe her feelings than her surroundings (which sort of kept being the same throughout the process). She told us that she was being betrayed by her husband at that time and killed by the villagers.

I was a bit shocked by how much it got her. You could see the fear on her face. At that point, I didn’t really wanted to go myself anymore.
After that was done, there was another process where you would accept what happened. So basically ‘forgive’ your past self. At that point everything seemed OK again.

After that another girl got a reading, she had a much more relaxing past life (in a cave eating a raw rabbit) which was also described much by feeling (she said “I can taste blood” before actively describing her surroundings) which made me more eager to try it myself again, since nothing really happened.

The third person was chosen in a different matter. Anita used the energy of everyone in the group with 1 chosen ‘medium’ (which was me) to ‘vote’ who would go next.
Eventually Anita was asking if I would want to go (which my unconscious mind would answer) and it said yes.  So here we go!

I was lying there, where 2 people before me had good and more fearful experiences. I was asked to focus on a giant gate where I would go trough, to find my past life behind that gate. The thing didn’t open for me, but I looked up. I saw something that resembles a hot air balloon. A moment later I was in some sort of shed, in a small village, it looked like the middle-east. It was very hot, there was a lot of sun as well. I saw a man in the shed that looked like me, but darker skin and longer hair. I knew it was me, not because I recognized him, but he had that look. That look what I have too when I’m working on a brand new startup idea. He had that too, even more than I had actually. In front of him was the balloon. It looked like a bad quality version of the ones today. The balloon was hard, made of leather, the basket was very badly made, you could see the floor trough it.

My shed was open, people walked outside, they didn’t really care about me, but did leave me alone. This was a comforting feeling to me, and I was happy.
So overall a great past life! Everything I describe, I said out loud while I was in my past life, so the other people could hear it.

I felt a handshake (like physical in my right hand) but I knew it wasn’t real. I focused on my right hand in my past life and a man was there. He didn’t speak but I could feel what he was doing there, we was a friend that helped me and came looking how it was going.

Anita brought me a bit forward in time. We were at some sort of smaller version of the Grand Canyon. There I was, with the man and some sort of wooden cart that I think carried the balloon. The balloon was in the center between us. I could feel the thrills.

Then the whole thing turned.

My past self was doing something, where I myself was thinking “am I really that stupid?”. I couldn’t believe what was going on. I didn’t want to describe it in front of everyone, without knowing if it actually was happening. I was dragging the balloon on top of a hill and jumped off of it with the balloon.

Then the images stopped coming. I couldn’t see what happened but I knew it very well. I died in the crash. I broke down because of this. My eyes were filling with tears, not because that I saw some sort of death, but that I was so ambitious there, real or not, it was a manifestation of me, that I wanted to give my own life for the balloon to make it work.

Moral of the story

After the past life workshop, I kept thinking about it. I saw it as a symbol. The balloon could have been anything else.
I am very ambitious, I don’t know how to give up, I failed a lot of projects and yet I keep going and keep learning and it never wears me down.
When is that too much? How long does it take for me to ‘crash’?

That was what kept me thinking.

The ending

The rest of the day was going really well, I talked to a few people, we still got a part of the ball pit set up!
When it was time to go, I arranged a lift to the station, and left with a whole bunch of new experiences, and thinking how I would write about them.

Conclusion

I started this 3-day series to encourage open-minded, smart people to organize meetups like this. Be yourself, get years worth of self-reflection in just a few days, and have fun while doing it.

I’ve been to a lot of meetings myself and the bottom-line is usually the same. Go do some networking, make sure you up to date with what’s happening in your industry etc. And while that is important, this will never let you be truly yourself around the people around you, even if they are like-minded.

You’ve read the last part of my story, let this be the start of yours!

Categories
All Hail the Pioneers Life & Low-Tech Uncategorized

All hail the Pioneers: Day 2

This is the second part of a 3-day series. Click here to view all posts in the series.

It was early in the morning. I just woke up expecting to be annoyed by the fact that I was sleeping in a tent in the middle of the woods (I don’t like camping in general) but that didn’t happen. I found the sounds of the early birds and even the moist air quite pleasant. I got up quickly and joined the rest of the people for breakfast. This didn’t last long, as I had planned to join René (A friend of mine who I knew before the festival) in making a hardware/software project that will allow easier networking between 2 people. It was my way of contributing to the festival, aside from smaller tasks.

It was some sort of ball pit where next to it a pillar with a big button was placed. Another pillar would be placed in the center of the festival. The idea was that you push the big button at the ball pit and a voice will tell you that you can enter the ball pit. The pillar at the festival would light up then, where, when pushed, another voice at the ball pit will tell you that someone is coming to join you. When 15 minutes have passed, the routine starts over.

I quickly grabbed 1 slice of bread, then we drove away to a working place not far from the festival.

At the working place

When I first entered the work place, an open space where different kinds of people go, it was heavenly. The whole thing was so peaceful, that even right now it still cracks my head why I liked this place so much. I think it must be the combination of the simple things that where there. The sound of the pool, the chickens that walked the lands, and the view of the big house that sat in the middle. Each one filling a gap that would otherwise be empty. Nothing overfilled, just enough to create a smooth surface ready to slide away on.

[easy_panorama id=”167″]

Although it wasn’t all that fun. I planned to go back to the festival around 16:00 as this ‘would be’ my last day. I would do some workshops when I got back (every day there are new workshops you can join)

We got delayed heavily on our project and ended up working for 8 hours straight if it wasn’t more. We couldn’t finish on time and we stopped around 22:00. While I did enjoy the view and the people at the working place I wasn’t that happy that I missed so much today at the festival. I decided to talk to my family and see if I can get an extra day so I can enjoy some of the workshops. After all my chores were delegated I could stay an extra day and as disappointing I was, as happy I am now.

We missed dinner, but we got some surprisingly good food (it was like chilli con carne with bacon) from the people who ran the working place. To my surprise, this was also my first technology-related conversation I had during the festival.

Back at the festival

We got back late. After a few talks I joined the campfire and got into a new conversation with ease. After I started talking to a good friend who just arrived, I felt like when I close my eyes, I would fall head-first into the campfire. So I decided to be honest and tell him how tired I was, instead of pretending to listen. We could finish it tomorrow.

I took a shower at the camping’s public bathroom which woke me up again. At this moment it was like 00:00 and I joined the campfire again. New people sat around the fire, and this is where the second and third conversation was about something in my comfort zone (entrepreneurship, technology). All other conversations were non-tech related. As I talked about my upcoming project/startup, my enthusiasm rose and thus did my energy level. At the time I really went to bed it was like 4:30.

But I was happy. I didn’t regret that I stayed so long at the workplace, as I would never had found such peace while working. I still got another day to really get into the festival. But that is for tomorrow!

Categories
All Hail the Pioneers Life & Low-Tech

All hail the Pioneers: Day 1

This is the first part of a 3-day series. To see all posts in the series, see: https://codebuffet.co/nl/category/all-hail-the-pioneers/

Here I am, sitting in the train, slowly descending back into civilization as I traveled from station to station, away from the Hamlet’s of the Netherlands, back into the cities. I’m looking outside, thinking about how I would write everything I’ve been trough in just 3 days. I think I got it now. So let’s begin…

First, I want to emphasize what this post is not. To all developers, investors, inventors, entrepreneurs and anyone else who has read some of my posts: This is not one of those posts. This is not a new project that I’m super stoked about, or a game that I love to explain every bit about, no, this is a story about how I got into a 3-day, medium-sized invite-only meetup called Permanent Beta Festival, set in the Netherlands, in a camping, in the middle of the woods.

I also do not represent The Permanent Beta organization in any way in this post, all the quotes, people and situations explained in this post, are the view of a single first-time visitor and volunteer of the Permanent Beta Festival.

This is also not a review. It’s not a compliment, it’s not critique either. Why not? Because when people read a review about something, they set, if they like the experience the reviewer had, expectations, mostly similar to the ones of the reviewer. If those expectations are not met, then the experience of the readers of the review is overall not very positive. Not because it isn’t, but their mind is locked on that one experience.

With our story it’s different. It doesn’t matter to me what your experience would be, just as it shouldn’t matter to you what my experience was.
Also reviews are typically about services like hotels, restaurants, and other places where there is typically a centralized ‘service’ that will make your experience as good as possible, either by having waiters, staff, and other people ready to serve you. They make sure you have fun. If you are not satisfied you go to the management and explain your complaints.

In the Permanent Beta Festival, everyone makes sure everyone else has fun. Imagine a place in the middle of the woods where there are tons of people who are like you, but total different interests. This is something I didn’t think was possible, but it was. I spoke with around 20 different people and around 3 of them were actually working in a tech-related field. So what did I and the other 17 people have in common? We were all, very, very crazy, out of the box thinkers. Every single one of them, including me, that was something that made me feel very comfortable in a very short time.

So let’s go back, before I was in the Permanent Beta Festival. How did it get started? I was working at home on a freelance project, when the phone rang, and I picked it up. It was a good friend of mine asking me if I could arrive earlier. Sure, I thought, because I expected the festival to start at Sept 27 for some reason. But then he asked, “Ok, can you come soon?”.

“…..”.

Alright. You know where this is going. I forgot I got my tickets for Saturday and Sunday at Sept 10.
I didn’t want to skip this event as I heard many great stories myself as well. So I moved my appointments, arranged a lift and left home nearly immediately, on the same day. It’s not a fun thing to do, but you should try it, it makes your brain strong to adapt to quick changes.

And away I went! Riding in a camper with a girl I just met at that moment. Around 3 hours later we were at the spot. At that time it was around 22:00 I think.
I immediately was astonished by the lights and the environment. It didn’t take me a long time to start my first conversation. It was about how to imitate a bagpipe. That’s all you need to know 🙂

What’s next

Tomorrow I will continue on the second part of the series. I hope you enjoy the read. Don’t be afraid to leave any comments that help me improve my writing. See you next day!

Links:

Permanent Beta Homepage: http://www.permanentbeta.nl/

Categories
Uncategorized

How very bad internet and enthusiasm made my game idea come to life in 3 days

You might think this is just a rant about a first-world problem, but most certainly not (well, maybe a bit, but the subject of this post is more about mental exercise through having limited resources)
I was on a holiday for 2 weeks in Stadtkyll, Germany with my family.

Before that, I played a card game called ‘Set’ where pattern recognition is a great skill to have to win. I’ll explain it quickly:

One person takes a card and puts it on the table (it’s the same person every time, no turns taken, and he can too play the game along with everyone else)
Each card has a color, shape and number (the amount of shapes). When you find 3 cards that have either one of the same properties, you yell ‘Set!’ and the 3 cards are yours. In the end, the person with the most cards win. I wanted to wrap this game logic in a simple, commercial package but didn’t know yet how.

Back to the holiday. Halfway-trough the holiday I had to update an Android app of mine. I needed to go into Google’s developer console. I did and my eye caught this:

Screenshot from 2016-08-15 12-33-59

And there we are: Peter in a enthusiastic state! You can wake me anytime for anything even remotely related to a Hackathon (not those awfully commercial ones, but the smaller ones are cool!)
It was august the 8th I think. I don’t know what happened that day,  but suddenly every creative moment of the last 3 weeks fell together. Boom. Finally a cool version of the Set-game arrived in my mind (for those interested in the game itself: I’ll leave a link at the bottom of the article).

I didn’t even read the form, that showed up after you hit ‘Submit your game’ – I just started coding asap. (Apparently it’s only for people living in Canada and US, which I found out afterwards :()
The idea I had was to make random bubbles with random colors and patters on the screen. On the bottom of the screen there would be a timer, and the example bubble(s) you have to find.
The timer would rapidly decrease from 100 to 0 in about 4 seconds. When you tap the right bubble, the points left on the counter will be added to your total score, waiting for a new game round.

But there was one problem.
The internet is very, very bad. Even the developer console took half an hour to load.
It was clear to me that I couldn’t use an engine like Unity or LibGDX (to which I’m used to make games with, especially when prototyping)

All I got was a text editor, a OS and some npm tools like gulp, harp (a static server that transparently compiles CoffeeScript and languages alike) and luckily Cordova, upon which I proceeded to write (or better say deploy) my game in. It was set (ba-dum-tss): It was going to be HTML5 and CoffeeScript (that compiles to JS) – No engine like Unity, no LibGDX, not even a JS engine!

But I didn’t want to re-invent the wheel entirely, so I decided to add a rendering engine library, called PIXI.js.
I started downloading PIXI.js (the minified version because the full version took too long to download). It was done in about 1 hour on around 200 bytes per second.

While this was downloading and failing a few times, I started to write the whole game’s logic in a CoffeeScript class with subclasses that represent parts of the game. It looked like this:

totalColors = 5
colors = [0xff4a4a, 0x3aef35, 0x4acaff, 0xefce35, 0xdb35ef, 0xd07925]

class Bubble
  # Bubble pattern and color here

class FindTheBubble
  gameOver: ->
    # Start gameover animation
    @status = 'gameover'

  startPlay: ->
    @status = 'playing'
    @points = 100

  constructor:
    # init empty variables, like array of bubbles and the total score
    @points = 0
    @status = 'idle'
    @bubbles = []
    for i in [0...10]
      # Add bubbles here
      @bubbles.push new Bubble(...)

    @bubbleToLookFor = random bubble of the list

  clickBubble: (bubble) ->
    if bubble is @bubbleToLookFor
      @total += @points
      # Start new game
    else
      #Instant gameover


  # This is what the future rendering engine will call each frame
  tick: (dt) ->
    if @status is 'playing'
      # Here I decrease the 100-points timer like this
      if @points > 0
        @points -= dt * 0.025
      else
        @points = 0
        @gameOver('time_up')

The result is that I can now play the game I had in mind using nothing but my browser’s console. This was not only quite funny to do, it also allowed me to tweak the logic, implement a different kind of score counter, and add more or less bubbles.

The finished game
The finished game

In the meantime PIXI.js finished downloading. I then couldn’t work much further because I never used PIXI.js. So I walked out of the bungalow and went to the park’s reception’s office where they have better internet (but still slow compared to what we’re used to at home). I cloned PIXI.js from GitHub so I had their docs.
I walked out and went back to our bungalow. It is quite a big walk, our house was on top of a hill, all the way on the other side of the park. It was a hill not made for nerd-legs. I don’t know how steep it was, but it felt like I almost had to climb back home.

With the docs under my belt, I started to write my class to a simple GUI. This is I think day 2. The GUI was just a bunch of colored squares and a score panel I made in Gimp.
It was really cool to see the GUI come to life without a single bug, since I already had tested my game with the class I made.

When I got home, I took off the rough edges: made sure the game scaled correctly on multiple resolutions (see this issue on GitHub), made the final bubbles, the screenshots and the entry on Google Play. And that was it! The game was born.

Wrapping up

It’s good to set yourself aside from the conveniences of today’s world from time to time. I’m sure i’ll do it again!

What happened here is that I learned to deal quickly with (intentionally) limited resources, and along the way, even made me think outside the box in no way I ever did!
I mean, I could have sat all day at the Reception’s office, and have a more decent connection, maybe develop the whole game like I usually do, and I’m glad I didn’t.

This experience was great for me, I hope you enjoyed reading this story, maybe learned something from it!

The finished game

It’s called Find The Bubble. It’s available on Google Play here.

If you have any feedback or questions don’t hesitate to leave a comment!

Categories
Uncategorized

I just made a fan-app for Pokemon GO!

Update: I removed the app from the play store because of the lack of users 🙁 I will bring it back soon, making it look more appealing 🙂

Hi there,

I’m a big fan of the Pokemon Go app and the community around it. I’m a big nerd, I admit, and I usually only go outside for my work, until this game got released :). My first Pokemon game was Pokemon Yellow on the Gameboy Color. A feeling like starting to play a game like that, never came back, until Pokemon GO came around.

What I would like to see was a more social aspect, something where you can make your own teams (in my app called Clans, to avoid confusion with Team Mystic, Valor, and Instinct), and catch together, even when you are apart.

You can see the pokemon you caught together, chat within the clan, and view user’s Pokemon GO activity.
All this stuff is not using any private API’s from Niantic but uses it’s own servers and database (you just upload screenshots to the app, and it can automatically detect what pokemon you caught), so it’s not illegal, or against Niantic ToS, and therefore doesnt have to be removed.

 

A version for Android is available here: https://play.google.com/store/apps/details?id=co.codebuffet.gogroup2
iOS is coming soon if there are enough users 🙂

Categories
OpenFire

Inside OpenFire: An overview on how it works.

The purpose of this post is to give insights about the workings under the hood of OpenFire and assumes you already know about it’s existence.
Never heard of OpenFire? See: http://codebuffet.co/2014/09/08/announcing-openfire-revolutionary-database-backend-server/ for a introduction.

How does it work?

I like to learn by example, and that’s what we are going to do here. The way this paragraph is organized is by showing an example code in OpenFire, then explain how it works under the hood.
This set of examples + explanation is divided in 3 tasks OpenFire does: Writing, Reading, and Syncing data.

For the next paragraphs, Suppose we start off with a HTML file like this:

<!DOCTYPE html>

<html>
<script src="http://openfi.re/openfire.js"></script>
<script>
// Intialize our DB
// tech-demo is our namespace for this project
var db = new OpenFire("http://localhost:5454/tech-demo");
</script>
</html>

This code will load the JavaScript SDK, set up a real-time connection with the server, and registers the client with the server, ready to send or receive any new data.

Writing data

Writing data in OpenFire happens with the ‘set’ command. Let’s imagine we want to register a user.

db.child("users").push().set({
  username: "peter",
  email: "peter@codebuffet.co",
  settings: { newsletter: true, rights: 'admin' }
})

Child brings us a path deeper, in this case, in the users path. See paths kind of like folders on your computer.
Push creates a new child and assigns a random unique ID, to make users kind of behave like a list.
Set writes down the object, overwriting anything within it’s path.
As you can tell from the example above, you can see that we are writing an object with 2 primitive types (strings, numbers etc) and 1 object (the settings object), which contains primitive types again.
After writing, our database looks like this (note the randomly generated id):

{
    "users": {
        "o1vwFb2qtotSV_Yukq+i8rKsiMtxn9OZUimZ": {
            "settings": {
                "newsletter": true,
                "rights": "admin"
            },
            "username": "peter",
            "email": "peter@codebuffet.co"
        }
    }
}

Users is now a rather small object, but you can imagine that with more than 100.000 users, the users object could become quite big. Fortunately, OpenFire flattens objects before being stored! The way this works is that objects and primitive types are split up in segments. Each object will be stored in it’s own path (think of the folders again like we talked about) I found this the best balance between scalability and storage efficiency.

Now let’s see how OpenFire stores our little Users objects:

OpenFire [Server] ->  Memory is now:
{
    "/tech-demo/users/o1vwFb2qtotSV_Yukq+i8rKsiMtxn9OZUimZ": {
        "username": "peter",
        "email": "peter@codebuffet.co"
    },
    "/tech-demo/users/o1vwFb2qtotSV_Yukq+i8rKsiMtxn9OZUimZ/settings": {
        "newsletter": true,
        "rights": "admin"
    }
}

The technology is called a path-value store. It’s a mix between a filesystem on your computer with folders and files, and a regular dictionary (key-value storage).
It allows ultra fast writes, because the server doesnt have to check if a parent object exists if you are writing a nested object.

Reading Data

When reading data in OpenFire, it will search for a object under the path what you are looking for, and, if it finds something, it will rebuild the object back to it’s original state. Let’s say you want to get the contents of ‘users/o1vwFb2qtotSV_Yukq+i8rKsiMtxn9OZUimZ’:

db.child("users").once("value", function(snapshot){
 console.log(snapshot.val);
})

The best way to explain this is by using this gif:

explanation1

It’s such a simple concept that brings tons of possibilities! Imagine it like folders and files on your computer again. Say you have everything about your users in 1 file, and you need to know the email of a user called ‘joe’. You would have to open the entire file in the memory of your computer, then start searching for this user.

Now imagine the same scenario, but then you got your users stored in folders. One folder for each user. Now you only have to open the folder belonging to joe, and open his file. This setup not only allows you to search faster, but it also takes less memory on your computer to do so. This is exactly what happens on OpenFire.

Syncing data

OpenFire sends data back and forth to it’s clients using real-time, serial data pushes. No HTTP API, no REST, no overhead.
Apart from using this to send and read data, you can also use it to sync in real-time. The way this works is by setting up a WebSocket connection, with a long-polling fallback for older browsers. This set-up happens as soon as you connect with your database. To sync with the database and other clients you subscribe to events.

db.child("users").on("child_added", function(snapshot){
 console.log(snapshot.val);
})

This code code will call the function each time a new child is added to the users object.

OpenFire keeps track on what objects users are subscribed to and how many. Using this knoweldge, OpenFire only checks objects for changes that are subscribed on. This makes the publish-subscribe system extremely fast, because no unnecessary loops and checks are done in the background. Objects that are not subscribed on are written directly, objects that are subscribed on, will only be checked on the places that matter. These subscriptions are stored in the so-called Meta Table. This is a special object, that exists in every OpenFire instance, that resides in /_meta in. the root of your database. It is the warehouse for server-related data like configuration and in our case the subscription to this event.

No one can write to this object directly, only the server can.

Wrapping up

I hope I have given enough insights to show how OpenFire works. In case you would like to know more, feel free to comment. I’ll be happy to explain more of it’s inner workings!

Categories
OpenFire Uncategorized

Announcing OpenFire Beta: Revolutionary database & backend-server

I’m a big fan of Firebase and services alike. I used it at the first hackathon I participated in. They make front-end development so easy and blurs the borders between development and production in such way that you don’t have to worry about it. Focus on the contents of your software!

For people unfamiliar with Firebase or noBackend in general, noBackend is the name for a trend where developers move a lot of their code normally residing on the server side to client-side tasks. It makes servers doing less heavy jobs, “borrows” their users PC to handle most of the tasks, and only do the absolutely necessary on the server (like security and storage)

Past week I have been trying to find open source alternatives on backend-engines just for fun and came across some good solutions like PouchDB, Deployd and Hoodie.
I really liked them, but they all were not 100% production ready, Hoodie doesnt have support for custom security rules, PouchDB (works on top of CouchDB) only allows user-private data with a custom database per user. And Deployd is slowly updated right now.

I was a little irritated at this, because I believe there is a lot more in the market for static web apps than we might think. A LOT more.

 

Draft5
Yellow, the OpenFire Mascot

Announcing OpenFire

Today I finally finished the OpenFire beta release. It’s done in 3 weekends. It’s far from perfect, still in beta but it’s awesome and I’m very proud of it, so proud I decided to open-source all the code and share the idea around with you guys. The purpose of this post is to gain initial feedback for me to decide how to go on with this project.

To begin, OpenFire doesnt want to be your database:
– It wants to be your backend, so you don’t have to worry about creating and maintaining one.
– It wants to interact with your database of choice, so you don’t have to.
– It wants to synchronize your data real-time across your users

All it takes is 2 commands to install and get started, if you like to check it out already, here is the link to our website: openfi.re

It is database agnostic

What did you say? Yeah! Soon, creating data static web apps will be just like creating data on a typical Django or Ruby on Rails app. You can just pick a database you like to work with and OpenFire will share it’s data with it.
This works by having a built-in abstraction layer, where you can make it work with other databases using plugins.

If you need speed to read/write, hook OpenFire up to Redis.
If you need to store large objects, just plug in MongoDB.

How awesome is that? Currently OpenFire only has a in-memory database that is gone when you close your server, but I will definitely make support for more databases. I first wanted a minimum viable product 😉

Real-time data pushes

The OpenFire SDK has built-in support to connect to the server using WebSockets or Long polling if WebSockets is not supported. Quickly listen to object additions, deletions or changes and receive data in a serial-matter for extreme speed that outperforms HTTP big-time.

Fast to read, fast to write

OpenFire has a number of home-made technologies that makes storage efficient and reading fast. For instance, if you store a big object, and only want to read back a child of that object, it will only load a part of the object in memory.

Contributions

CodeBuffet is a 1-man company, so all contributions, big or small are openly respected and encouraged! I currently haven’t got the time to write good documentation, but I will definitely do that somewhere in the weekend!

Roadmap

The reason for me releasing this project already is to gain initial feedback and to see if more people like the idea.
OpenFire is definitely not production ready yet. It lacks certain features for running a stable server, like wrappers for a proper database like MongoDB or Redis, also there are no security settings yet for you to configure.
I’m planning to use OpenFire long-term for my next projects and can’t wait to see it production ready.
The following features are planned short-term:

– Database support (I start first with MongoDB I think, as it’s the most general-purpose database suitable for this kind of work)
– Offline compatibility in the SDK
– Security (validate data before storing in the database)

Get started

Our website is at: openfi.re
You can play with the example or download it for yourself 😉

How does it work?

Click here for an in-depth overview.