How to Ink Annotations in UWP Apps

Many of us have a habit of writing notes while reading. While I was working on the article about Editing Annotations, I was wondering whether whether we would like to do the same while reading charts. Would a chart reader want to annotate the chart by drawing on the screen to share his/her own thoughts? The ability to write and draw directly on our computer and mobile screens can personalize the experience for the chart reader, offering a deeper level of understanding and analysis.

Suddenly I had multiple different questions! I wanted to know if FlexChart could let users do the same. As I started to experiment with FlexChart, I came across Microsoft's support for Windows Ink in UWP apps.

In this blog, we'll walk you through how to combine Windows Ink with FlexChart -- to write and draw annotations on the chart's surface!

We'll discuss:

• Inking Basics
• Adding Ink support for FlexChart
• Analyzing ink for understanding user input
• Converting inked notes to Text annotations
• Converting inked drawings to Shape annotations
• Styling Annotations using the ink’s drawing attributes

Inking basics

The Windows Ink Platform is made up of four different components that allow you to add ink to your apps:

• InkCanvas: A XAML UI platform control that, by default, receives and displays all input from a pen as either an ink stroke or an erase stroke.

• InkPresenter: A code-behind object, instantiated along with an InkCanvas control. It provides all default inking functionality exposed by the InkCanvas, along with a comprehensive set of APIs for additional customization and personalization.

• InkToolbar: A XAML UI platform control containing a customizable and extensible collection of buttons that activate ink-related features in an associated InkCanvas.

• InkD2DRenderer: Enables the rendering of ink strokes onto the designated Direct2D device context of a Universal Windows app, instead of the default InkCanvas control. This enables full customization of the inking experience.

We’ll be using the InkCanvas and InkPresenter in this blog to ink our annotations. Before proceeding, ensure your system meets all the prerequisites listed in this article.

Adding ink support for FlexChart

Using ink with FlexChart is a simple process. To get started, to place an InkCanvas object in the page that contains FlexChart.

        <Grid x:Name="chartContainer">
            <Chart:C1FlexChart x:Name="flexChart" />
            <InkCanvas x:Name="inkCanvas"/>
        </Grid>

The InkCanvas by default supports receiving inputs only from a pen. For supporting inputs received from a mouse or from touch gestures, we would need to configure the InkPresenter that is exposed by the InkCanvas. The following is an example of the InkCanvas accepting inputs from pen, mouse and touch gestures:

public MainPage()
        {
            this.InitializeComponent();
            inkCanvas.InkPresenter.InputDeviceTypes = 
                                            CoreInputDeviceTypes.Mouse | 
                                            CoreInputDeviceTypes.Touch | 
                                            CoreInputDeviceTypes.Pen;

        }

As the InkCanvas is basically a XAML UI control, overlaying it over FlexChart would prevent our interactions with the chart. To overcome this issue, we can manipulate the ZIndex of the InkCanvas bringing it to the front when we want to draw or send it back otherwise.

if(allowDraw)
                Canvas.SetZIndex(inkCanvas, 2);
            else
                Canvas.SetZIndex(inkCanvas, -1);

Adding InkCanvas and configuring the InkPresenter marks the end of the first step for inking annotations in FlexChart. The next step analyzes the ink strokes to understand whether the input was text or shape.

Windows Ink - analyzing ink

While analyzing ink may sound like an extremely arduous task that involves prior knowledge of handwriting and/or shape recognition, it isn't! Windows Ink provides a huge set of APIs in the form of the Windows Ink analysis engine. This helps to classify, analyze and recognize a set of free-form ink strokes on the InkCanvas.

From this large set of APIs, we want to begin the analysis process with the InkAnalyzer. The InkAnalyzer primarily takes in a set of ink strokes to process, performs the analysis and classification into shapes and text, and provides the results. We’d recommend you go through this article for more details on the ink analysis process. The following snippet shows an example of the process:

var analyzer = new InkAnalyzer();
var strokes = inkPresenter.StrokeContainer.GetStrokes();
if (strokes.Count > 0)   //Strokes to process
{
analyzer.AddDataForStrokes(strokes);
var analysisResult = await analyzer.AnalyzeAsync();
if (analysisResult.Status == InkAnalysisStatus.Updated)
{
//New Strokes have been identified as text/shapes.
}
}

After the analysis has been completed by the InkAnalyzer, the next step finds the exact texts and shapes and converts them to the appropriate chart annotation. Before we start the next step, it is important to understand how the InkAnalyzer processes the ink strokes.

Understanding the InkAnalyzer

The InkAnalyzer differentiates and recognizes the categories of ink input as nodes which are identified by the InkAnalysisNodeKind enum. Inputs recognized as a single word of text are categorized as ‘InkWord’ while inputs recognized as shapes are categorized as ‘InkDrawing’. In the next sections we would see how this categorization of the node kind will be used.

Converting inked notes to text annotations

The process to convert ink strokes to annotations begins after the analysis process has been completed. We call the FindNodes method for InkAnalysisNodeKind.InkWord to get all the ink strokes that would represent text annotations on FlexChart.

if (analysisResult.Status == InkAnalysisStatus.Updated)
{
//Identify words 
words = analyzer.AnalysisRoot.FindNodes(InkAnalysisNodeKind.InkWord).ToList();
}

The FindNodes call above returns a collection of InkAnalysisInkWord objects each of which provides the recognized text, as well as the bounding rectangle that contains the ink strokes. We then use this information to process and convert an InkAnalysisInkWord to a Text annotation as shown in the below code snippet.

private Text ConvertToTextAnnotation(InkAnalysisInkWord word)
        {
            var rect = word.BoundingRect;
            var x = rect.Left + rect.Width / 2;
            var y = rect.Top + rect.Height / 2;

            var textAnno = new Text(word.RecognizedText)
            {
                Attachment = AnnotationAttachment.Absolute,
                Location = new Point(x, y)
            };
            textAnno.Style = //get annotation style 
            return textAnno;
        }

Converting inked drawings to shape annotations

The method to find the shapes drawn on the chart is similar to the one we used above for identifying words. We call the FindNodes method for InkAnalysisNodeKind.InkDrawing to get all the ink strokes that would represent shape-based annotations on FlexChart.

if (analysisResult.Status == InkAnalysisStatus.Updated)
{
//Identify shapes 
drawings = analyzer.AnalysisRoot.FindNodes(InkAnalysisNodeKind.InkDrawing).ToList();

}

The call to FindNodes above returns a collection of InkAnalysisInkDrawing objects. Each InkAnalysisInkDrawing object tells us about the shape that it represents via its DrawingKind property and you can easily see whether the drawn shape was a circle, square, ellipse, etc. While the InkDrawings that represent a circle, square, rectangle or an ellipse can be directly mapped to their corresponding annotation object, the other shapes can be mapped to a Polygon annotation. For example, the following demonstrates converting a circle shaped ink drawing to a Circle annotation:

InkAnalysisInkDrawing drawing;  //The ink drawing object to convert
var rect = drawing.BoundingRect;
var x = rect.Left + rect.Width / 2;
var y = rect.Top + rect.Height / 2;
if(drawing.DrawingKind == InkAnalysisDrawingKind.Circle)
var circle = new Circle(text)
{
Radius = rect.Width / 2,
Location = new Point(x, y),
Attachment = AnnotationAttachment.Absolute
};

The sample attached at the end of this blog shows how you can convert other shapes to annotations in a similar fashion.

Styling annotations using the ink’s drawing attributes

So far, the annotations that we had converted were devoid of any styling attributes. We can add styling by taking advantage of the drawing attributes that are exposed with the ink strokes. Each ink stroke that’s drawn on the InkCanvas has certain attributes related to its drawing such as the color of the ink stroke, thickness of the pen, etc. To add styling to the converted annotation, we can define a new ChartStyle that uses information provided by these drawing attributes.

The following snippet and image show an example for the same:

IInkAnalysisNode node;   
var strokeId = node.GetStrokeIds().First();
var stroke = inkPresenter.StrokeContainer.GetStrokeById(strokeId);
if (stroke != null)
{
var color = stroke.DrawingAttributes.Color;
var thickness = stroke.DrawingAttributes.Size.Width;
var style = new ChartStyle();
style.Stroke = new Windows.UI.Xaml.Media.SolidColorBrush(color);
style.StrokeThickness = thickness;
style.Fill = new SolidColorBrush(Windows.UI.Colors.Transparent);
annotation.Style = style;
}

This completes the process of inking annotations to FlexChart. You can also add interactions to the chart by including the EditableAnnotationLayer. This would allow the end-users to select, move and delete the converted annotations providing a more powerful user-experience to them.

Download the FlexChartInkDemo