Monogame – Working with Touch

*NOTE – This may also work on other platforms with touch as well, but this hasn’t been tested on anything other than Android at the current moment.

One fundamental aspect to understand about touch and gestures is that code needs structured around the idea of continuous input. Thus, any touch recognition code should be encapsulated somewhere within the main game loop; where this is would be dependent upon how one elects to parse input. The reason for this is that the nature of all touch processing is a sequence of touch events that get fired in succession; there are only a few exceptions to this rule.

The other aspect to understand clearly is nomenclature mapping. What you think you want to achieve in your design may, unless you’re already familiar with either the Monogame or Xamarin frameworks, not necessarily be what either of those aforementioned frameworks are calling it through its API parlance. So be prepared to step outside your normal wheelhouse and, perhaps, discard some long-standing assertions about what truly a touch/gesture really is.

First, the consideration needs made as to if you’re going to handle touches or gestures – don’t get too caught up on thinking that a touch is a single press, because this will cause you some unnecessary grief later. If your program ebbs toward the former, there’s usually no requisite configuration needed, other than to check for the existence of a touch-capable input device and adjust your engine accordingly (the only other exception here would be if one wishes to perform touch-to-cursor mappings, but this is outside the immediate scope of this article). Conversely, gesture handling requires some configurations be made before they can be used, and this is where the Monogame documentation, as well as a majority of the information available on various forums, falls fatally short. We will focus on this once we start addressing gesture handling, which follows a look at touch handling.

Checking for the existence of hardware with touch capabilities can be queried through the TouchPanelCapabilities object, triaged through the TouchPanel static object as such:

TouchPanel.GetCapabilities ().IsConnected;

The IsConnected property is a boolean that indicates the presence of a touch-capable device. Obviously, the absence of one suggests either that the device is intermittently available, or that other sources of input are necessary. In this case, neither the touch nor gesture APIs would be valid.

Concerning touches, the current state of the touch panel holds a collection, called TouchCollection, of TouchLocation objects. Each of these correspond to a single touch event generated by the user when a press is determined to have occurred on the sensitive surface. It’s fair to suggest that multiple points of contact yield multiple TouchLocation instances within the TouchCollection (i.e. as a practical example, one finger would generate one instance, two fingers would generate two instances, etc. This can best be observed while debugging a program using this technique, and paying attention to the Id property of the TouchLocation instance(s).). In order to properly ascertain the most accurate state of the touch panel, we’ll need to do the following:

  1. Obtain the current state of the touch panel
  2. Determine if there are any TouchLocation instances in the TouchCollection collection
  3. If there are, take relative action based on any of the properties of the current TouchLocation instance

This can be accomplished using the following general template

TouchCollection tc = TouchPanel.GetState ();
foreach (TouchLocation tl in tc) {
 // Do something here
}

Fortunately, TouchLocation instances are very simple to use. There are only four properties that are of any practical significance, and one utility function for trying to determine the previous location (which, at least during my testing, wasn’t useful for anything). They are as follows:

  • Id – An Integer that uniquely identifies a single TouchLocation instance within a TouchCollection. It’s unknown how its potential values are determined, or if orphaned values are subject to recycling after the host TouchLocation has fallen out of scope.
  • Position – A Vector2 that provides the X and Y coordinate serving as the source for the event. Effectively, this is the location on the sensitive surface where contact was made.
  • Pressure – A floating-point number indicating the pressure of the touch. I’m unclear on how exactly this works, since my tests always reported a zero in this property. The only conclusions I can come up with here are either that my device doesn’t support touch sensitivity of this kind, or I missed a configuration step to enable this functionality.
  • State – An instance of TouchLocationState that determines what kind of touch we’re dealing with. This property can have one of four values:
    • Invalid – The press has been somehow deemed invalid; I’ve never seen this occur in testing, and am left to think that either I failed to satisfy conditions to make this occur, or that it’s simply a dump case for an exceptional condition.
    • Moved – Seen when a press has either remained in the same position or has moved. This is a very important state as it has a great deal of practical application, so do try to keep it in mind.
    • Pressed – Observed when a point of contact on the sensitive surface is newly minted. This is only fired once, and will not be seen again, regardless if the contact remains afterward. This could potentially be a source of confusion for a number of programmers.
    • Released – Likely the last state that a TouchLocation would be in before falling out of scope, it will be fired when the source of contact which generated this particular instance is no longer available.

Having said all of that, you now have enough information to properly utilise touches within your programs. As was stated before, simply ensure that your code is contained within the main game loop somewhere, since the data will likely change per frame, or at least as often as it can. An example of how to implement code based off this logic will be illustrated at the end of this discussion.

Gestures, insofar as Monogame is concerned, are a bit of a hassle to work with, mostly due to the lack of upstream documentation. We will seek to correct this imbalance, if not for the official documentation, then at least for programmers who wish to learn more about them.

As was stated previously, although Monogame performs a considerable amount of work to make available the touch and gesture API for programs deployed on capable hardware, gestures are left a bit open-ended by comparison to their touch counterparts. What this means is that as a programmer, you’re required to provide some configuration before the gesture API can be utilised. Here, we assume that you’re only interested in gestures, and not touch-to-cursor mappings, hence will only discuss the former.

Before proceeding, some basic information should be given as to the nature of gestures, and how they’re procedurally handled.

Although there is some very minor overlap with how a touch and gesture are expressed, they are two discrete entities. Gestures can be composed of one or more points of contact, and it’s expected that the locations of these contacts will change, in a particular way, over an indeterminate amount of time (ergonomic measurements would likely dispute this generalisation, but it is for this conversation a generalisation, and not a scientific claim). The ways in which these contacts change, or at least the resultant shape the change yields, as well as the number of contacts involved in the measurement, hints at the particular kind of gesture. In other words, a single point of contact that has a gradual change in its X-axis, be it positive or negative, which would yield a ray (or a series of small rays), is generally considered to be a horizontal drag gesture. When considering the same principles but looking to the Y-axis instead, we now find ourselves dealing with a vertical drag gesture. Pinch and zoom gestures typically involve two or more points of contact that move near concertedly away from or toward each other along the same logical axis. Perhaps paradoxically, at least when considering a contrast between touches and gestures, taps, double taps, and long-presses are registered as gestures as well; these are more concerned with the sustainment of a single point of contact relative to the time from when it was first recognised.

From a stock perspective, Monogame provides eleven types of gestures, referred to as GestureTypes. These types will effectively determine how gesture detection is performed (it’s unclear if the GestureType framework can be extended to facilitate custom gesture types, but this is a considerably advanced topic which will not be discussed here). However, Monogame will not automatically read the touch panel for gestural input. Instead, it needs instructed on which kinds of gestures to detect, and this is provided by the programmer. In any non-looping block of code, preferably during the initialisation routines, you’ll need to specify what are called EnabledGestures, which is a property of the TouchPanel static object. Multiple gestures can be configured by OR’ing one or more of the types together in the assignment statement. For example, if I wanted to parse for both a HorizontalDrag and a DragComplete gesture, I would write the following statement:

TouchPanel.EnabledGestures = GestureType.HorizontalDrag | GestureType.DragComplete;

Once this is complete, you’ll have done enough to get Monogame to start playing nice with at least these two kinds.

Parsing gestural input is, in essence, no different than parsing touch input, but there are some minor differences to the process. To start, we must first determine if there are any gestures with which to read data from. If we do not do this, attempts to read directly from the gesture store will generate fatal exceptions. Fortunately, the TouchPanel static object provides a boolean property called IsGestureAvailable, which will inform clients of the availability of queued gesture data. If we have data, we must convert the data into a sample, which is packaged into the GestureSample class. As with the TouchLocation object, the GestureSample object contains several properties that are of practical interest to the programmer, especially when making contextual decisions that respond to this kind of input. GestureSamples include the following properties:

  • Delta – A Vector2 instance which provides the delta, or difference, data for the first touch point in the gesture. This will change over time, and will always be relative to the coordinates where the touch was first recognised.
  • Position – A Vector2 instance which contains the current coordinates of the first touch point in the gesture.
  • Timestamp – A TimeSpan instance that indicates the time when the gesture was first recognised.
  • GestureType – A GestureType instance that indicates what type of gesture was determined based off several criteria.

Additionally, GestureSample contains properties called Delta2 and Position2, which are used to track a second point of contact that’s being measured as part of the current gesture. What this implies is that insofar as the stock gestures are concerned, Monogame will only be able to handle gestures where no more than two points of contact are involved.

My advice here is to experiment with the data through debugging until you’re comfortable with how these gestures are read, because there are some nuances with how the data continuously polls respective to different gesture kinds. For example, a HorizontalDrag gesture will, while the drag is occurring, constantly emit the HorizontalDrag signal until the contact source is released, terminating the gesture. At this point, if one is checking for the DragComplete signal as well, releasing the contact source will cause the touch panel to emit the DragComplete signal.

Examples:

To determine if a single press has been made:

TouchCollection tc = TouchPanel.GetState ();
foreach (TouchLocation tl in tc) {
 if (TouchLocationState.Pressed == tl.State) {
  // Execute your domain-specific code here
 }
}

To determine if a press was made, and has been held in the same position for an arbitrary period of time:

TouchCollection tc = TouchPanel.GetState ();
foreach (TouchLocation tl in tc) {
 if (TouchLocationState.Moved == tl.State) {
  // Execute your domain-specific code here
 }
}

To track the position of a horizontal drag:

(1) During game initialisation:

TouchPanel.EnabledGestures = GestureType.HorizontalDrag;

(2) During game loop:

while (TouchPanel.IsGestureAvailable) {
 GestureSample gs = TouchPanel.ReadGesture ();
 if (GestureType.HorizontalDrag == gs.GestureType) {
  // Execute your domain-specific code here
 }
}

Vagabond 1.0 – Nearly Here

Yes that’s right. After much toil, and missing my self-imposed deadline for a release, I’m nearly complete with the program and can give it a 1.0 which means that it’ll go into General Availability.

I’m not posting Vagabond News on its website because I want that strictly to be a place to get the program from and a reference for information that developers would be interested in. Just wanted to get that out of the way as well. 🙂

Right now the program is about 98% done. All that remains is to test two additional features, push to the repository (assuming they work, which they will) and then build Debian, RPM and Source Taballs for deliverables. I haven’t built Debian packages for quite some time so I’m having to go through several refreshers to bring myself back up to speed on the workflow.

After this release, I have some additional features I want to work in so development will continue alongside general maintenance coding. I want to be able to add support for choosing to use either the official Android SDK or to use the Replicant SDK. This is more of an ethical decision as I feel like Replicant is a great choice for developing in an open-source manner but might not be the best idea if you’re writing Android to be published on Play. But the option should be there. I’m also considering the idea of implementing SDK Administrative Task Shortcuts. These are commands that can be passed to Vagabond that will inflate to more robust tasks passed to the VM that maintain the SDK.

If all goes as planned, and work and life leave me alone long enough, I can potentially have this ready to go by the end of the week.

Vagabond – Update

Aside from the getandroidsdkdl script that I was offering for people who wanted to get started with Android Development on Linux, another cool tool I wanted to throw out there was a script I call “Vagabond” which is a Vagrant deployable which automatically configures a development environment that you can use for Android Development on your Linux computer. In fact, it may prove far easier to use it instead of the getandroidsdkdl script (it actually builds off of it). The actual Vagabond script isn’t available yet but I’m letting a rather close circle of friends test it out first before I make it public which shouldn’t be too far off from now. I wanted to give an overview of what it does.

If you’re unfamiliar with Vagrant, follow the previous link to get a better understanding of what it is.

What is Vagabond?

Vagabond is little more than a bash script that you’ll download and execute. Once you execute it, all you’ll need to do is sit back and relax while it configures the VM, installs all of the requisite packages to both update the base VM and the required packages for the Android SDK, and then exposes the synced directory from the guest to the host via host’s PATH and that should take care of everything. The user can then install an IDE of their choice on the host and run the Android SDK tools from their host computer.

Got that? To further simplify, the purpose of the Vagrant image is to host and isolate the the Android SDK and to make it even more portable since it’ll exist exclusively in a VM. The Vagrant Provisioner Script, in tandem with the Vagabond Script, exposes the SDK hosted in the guest VM to the host machine. If it still doesn’t make any sense, you may need to read up on what Vagrant is so that it sinks in a little more.

Vagabond Usage

Vagabond is, and will continue to be, stupid simple to use. I’m planning on adding some additional features to it but for right now, it only requires that you provide it with a directory where you want Vagrant to install and configure the VM in. That’s it. The rest is up to the script.

vagabond.sh [vagrant-project-directory]

Vagabond Details

Vagabond relies on three different pieces of software – Vagrant, Git and Zenity. It will check for the existence of these three on the computer before attempting to do anything. If either one of these are missing from the computer, the script will fail and return an error code specific to whichever program was missing. These codes are detailed in the comments in the script.

Clearly, the reliance on Vagrant should be rather obvious. Git needs to be installed since Vagabond will pull down the Vagrantfile and bootstrap provisioning script from my GitHub. Zenity is used to display the output from Vagrant while it’s running the VM configuration and provisioning script. The choice to use Zenity was made since the output from the provisioning script couldn’t easily be piped back to the terminal that was running Vagabond. I’m still not really sure why this was the case but that’s what was happening. If I can figure it out, or if someone a bit more skilled at bash scripting than myself modifies the source, I’ll yank the Zenity dependency out of there. I can’t say I don’t entirely dislike it though.

Vagabond then attempts to make the directory that the user specified while launching the script. No real magic here. The only thing is that it will check to see if the directory exists prior to running mkdir. Some may see it as a redundant check since mkdir will cater to already existing directories but I wanted some conditionally verbose output. Sue me.

Next Vagabond will move into the new directory and use Git to clone from my GitHub repository that contains the Vagrantfile and bootstrap Provisioning Script. Details on the Vagrantfile can be found if you read the Vagrant documentation. The purpose of the bootstrap script will also become evident as well. At least what it does with regard to Vagrantfile. Cloning from the GitHub will make a vagabond directory inside the directory the user specified for the Vagrant Project and we don’t want that. Vagabond will move the cloned files out of the cloned vagabond directory into the Vagrant Project directory and then remove the cloned directory.

At this point, we’re ready to use Vagrant. Now that Vagrantfile and bootstrap are in place, a call to vagrant up will use the Vagrantfile to configure the VM. Vagrantfile will then rely on bootstrap to supply additional configurations and instructions to the VM. The Vagrant Box used is the hashicorp/precise32 (Ubuntu Precise 32-bit). All other Vagrant configurations are defaults. The bootstrap Provisioning Script does the following:

  • Update apt repositories
  • Upgrade the system as well as install base dependencies for the Android SDK
  • Use a modified version of my getandroidsdkdl script to download the Android SDK from Google. This script will also unpack the SDK into /opt in the VM and configure permissions on it.

Once this is done, Vagabond will expose the synced directory, which is linked to the directory in the VM that contains the SDK, to the host system via a modification to PATH. That’s it. After that, so long as the VM is running, the host will have access to the SDK. The user can download and use whatever tools they want. Since the SDK exists exclusively in the VM, you can port the VM to anywhere you want.

What’s Next?

As I mentioned before, I’m letting a few people test this before I give it GA. Once I get those test results back and make some modifications to the script based on those tests, the version will be locked as 1 and released for GA. Stay tuned!

Development Journal – Smsr Update 001

I’d like to think that I started working on a rough draft with Smsr and then it evolved into something else.

This “rough draft” had a bit of code in it encapsulated in an entity called ConversationThread. A function of this class in particular was really nasty since what it did was take all of the SMS messages on the device and effectively organize them in a cohesive manner that didn’t rely on querying sub-providers like Sms.Inbox or Sms.Sent. The reasoning behind this was that when I would query some of the more specific Content Providers, like Sms.Conversations, the Cursor returned from that query would only contain three fields regardless of the applied projection. In other words, despite the fact that the contract class inherited from base interfaces that gave it the additional fields you’d want, this inheritance doesn’t apply to what’s returned from queries against the provider because these columns are missing from the Cursor. So it wouldn’t contain things like ADDRESS or _ID. And because these Content Providers weren’t designed with the ability to perform SQL Joins via queries (despite the fact that the documentation would lead you to believe so), the only other alternative, really, is to perform several separate queries against several different Content Providers and join them using either MatrixCursor or CursorJoiner. The issue there is (A) how to do this without blocking on the UI thread and (B) because subsequent queries would rely specifically on information that was contained in previous queries, how then to make sure that the queries had completed before attempting to use them?

I really tried to avoid the solution where multiple queries were involved because it just seemed so… lackluster. Not to mention that the idea of tossing back and forth between five or six Cursors seemed like an explosion in the works. But there was a major issue with the way I was doing it originally – real-time updates. Because the backing data was effectively abstracted into this seemingly convoluted construct that didn’t directly bind to the backing data, if something changed in the data (i.e. received a new text message), the structure would effectively have to be completely rebuilt. One pass on building that thing was expensive enough since it relied heavily on Collections. Doing it over and over again could lead to some really janky issues.

So I went back to the drawing board. And what I came out with, after struggling for a bit to understand MatrixCursor, was a solution that does in fact query multiple Content Providers which will yield several Cursors and then combines them into a MatrixCursor which is then given to my custom CursorAdapter. The result here also adds to a modified layout for the Conversation Stubs which uses a LayerDrawable to get the contact’s image (default if none) and apply a gradient over top of it that leads to the other half of the list.

Unfortunately, this change has broken the Activity that will show the Conversation Thread because the data that’s now contained in the stubs is different than what that Activity was looking for in the first place. But the stub viewer looks nice! 😀

If you want to follow along, the branch that this work is being done on is the “rewrite” branch.

https://github.com/gregfmartin/smsdemo/tree/rewrite

Update

Kid-rows! Don’t ask me what that means exactly because I don’t know either. I just made it up.

I’m still plugging away at apps. Smsr is in the process of undergoing a major rewrite. Last weekend I was playing with the SMS Content Provider again and having found some glint of genius in a single moment or realisation, I found a significantly more efficient way to write how it works. Implementing it throughout the project will effectively result in an entire rewrite but what’s computer science without the “science?”

Even cooler than that, my talk proposal for the Ohio Linux Fest (OLF), “Android Development on Linux,” has been approved and I received my invitation yesterday morning (read date on this post). OLF takes place in Columbus, OH starting on October 24th and runs the whole weekend until October 26th at the Columbus Convention Center. There are already a lot of great talks scheduled for there and my friend Brian Wagner is also giving a talk there about DevOps as well. If you’re planning on attending the event and want to attend either of our sessions, both Brian and I are going to be giving our talks on Saturday the 25th. Room placement and times are still tentative at this point as the event planners are still going through everything. It’s shaping up to be a really awesome event so if you’re into technology or open-source and you’re in the area, make sure to stop by!

Development Journal – Smsr

The SMS client that I’ve been working on for Android is now being renamed to Smsr (even though the root directory of the project will still say sms_demo).

I’ve uploaded the source code to Github so feel free to take a look at the source, fork or do whatever the hell you want with it. I’m going to be updating the contents of the wiki and the project landing page as major changes are made to the upstream build.

http://gregfmartin.github.io/smsdemo/

Development Journal – Android SMS App

For the longest time, I’ve been wanting to write a SMS app for Android. The issue originally (and still is unfortunately) was that if you’re running a device that has a version of Android less than KitKat (4.4.x), a programmer doesn’t have access to native public-facing APIs to get access to these Content Providers. Instead you have to rely on the Reflection APIs to poke around for them and, if they’re there, use them that way. It might not seem like it at first but Reflection can be a bit heavyweight on more resource-constrained devices (which can potentially toss ANRs or even cost battery cycles) and also makes for some really nasty looking code. I’ve worked on several projects where Reflection was the order of the day (especially for things like accessing Enterprise Configurations in WifiManager) but unless I really had no other choice, I tried to stay away from the Reflection solution as much as possible. Even going to the lengths of not writing this SMS app I wanted to do for so long because there wasn’t a public API for it. Enter KitKat with these packages and now I’m going to town.

Putting aside the fact that Google likes to arbitrarily break your code with new upstream versions, the package that contains the contract classes for these Content Providers, android.providers.Telephony, is actually quite solid. Unless they go ape-shit on this package like they did with ContactsContract (which is an absolute fucking nightmare to work with), we should be okay for the time being.

I’ll save the technical discussion for another post (or even a video) but what I wanted to do here was showcase the current state of the build that I have for the SMS app. Honestly I’ve spent more time trying to figure out the nuances of the contracts and how to use them and fucking around with 9-Patch than anything else (9-Patch is cool technically but why can’t someone at Google make decent documentation?) Additionally, I wanted to provide a tentative roadmap for where this is going.

A listing of all Conversation Stubs
A listing of all Conversation Stubs

Here’s a screen capture of the home screen which will display what I refer to as the “Conversation Stubs.” No, those aren’t native constructs in Android, it’s a custom object that I’ve built for this app. The goal here was simple – after doing all the magic necessary to sort the messages to have them make sense, grab the most recent one and from it derive the contact’s display name and a 45-character snippet of the message body. If the snippet length equals 45-characters, an ellipsis is appended to it to indicate that there’s more to the message than what’s visible. What’s missing here is an indication that the message was sent from you to the intended recipient (coming in the next nightly build). By the way, I swear I didn’t plan the colour of the ActionBar to match that of my site. 😉

Pressing on one of the Conversation Stubs will bring you to another screen which will display the entire Conversation Thread.

A Conversation Thread listing
A Conversation Thread listing

As with Conversation Stub, a Conversation Thread is not a native construct to Android. Now if you’ve done a little digging into the package yourself, you might be thinking why not just use the Conversation Contract? I’ll do more on that in the technical discussion.  Although there are no clear visual cues as to who said what, messages that have been sent from you to the recipient are on the right side and messages from the recipient to you are on the left. Each message is time-stamped currently using 24-hour format (this will be customisable in a later build). The thread recipient’s display name occupies the ActionBar in the top-left. The list of messages is sorted in descending order from the top and will automatically start from the bottom when opened which will display the most recent messages. Right now, there’s not much else here other than what you see.

 

Current, if short, Preferences
Current, if short, Preferences

The last point of interest here is the slowly building Preferences. One of my targets here is to have a decently high level of customisation while still retaining the overall look-and-feel of the app that I want. The bottom option is the more interesting one right now. As I mentioned before, KitKat will allow a user to specify an app to be the dedicated Messaging app. Of course I want the user to be able to choose this one as the default if they wish to do so. Pressing this brings up a dialogue that will allow you to choose the default app.

Overall, the goal here is quite simple. I prefer as vanilla of an experience on my Android devices and I like the Google Play Services and default apps but I have a serious beef with Hangouts. It just feels too clunky and I wanted something a little more back-to-basics. The current build cannot actually compose or send SMS yet (coming in two builds) nor can it determine or store draft messages (coming in a build after the previously mentioned one). MMS is also not yet handled but is coming shortly. I do not want to do anything fancy like integrate with Google Accounts like Hangouts does. I want this to be a pure unadulterated SMS/MMS experience. Nothing more.

Some tentative features that I have in mind though are sticky messages, cherry-pick deletion, hidden threads, export a conversation thread to a file (I’ve actually had a lot of interest in this from some clients for legal reasons) or print a conversation thread directly to a printer (relies on Android Print Services) or email a conversation thread to an email address, bulk messaging (one-to-many) and group messaging (many-to-many).

I’ll do a technical discussion on how to use the SMS Contracts and all of the nuances that I’ve discovered with them soon. Either way, I’m looking to have a first version published by the end of the month.