Moving Google Drive Docs/Sheets/Forms between workspaces/organizations

Recently, I needed to move a complicated Google Form and an associated Google Sheet that stored the responses to another account. Within an organization, this is pretty easy to do, but moving between workspaces doesn’t appear possible.

There is a workaround by using a Google Shared Drive in the target account and adding the source account to the Google Shared Drive. The source account can then move the files to the Shared Drive and doing so transfers the ownership of the files to the Shared Drive. At that point, the target account can go into the shared drive and move the files to their drive.

Jumping through all of these hoops preserves existing links to the form and avoids the need to relink the responses sheet which can sometimes be problematic.

Onboarding new users with GAM with an assist from CloudHQ

When someone is hired at my school, our HR system sends tech support an automated message with some basic information (first name, last name, start date, personal email address, job title, supervisor, etc.). Unfortunately, this isn’t formatted the best and I used to copy/paste the information from this email into an onboarding Google Sheet I used to mange the process.

I decided to try to improve on this process. I wanted the email to automatically go into my Google Sheet. While I’d love it if our payroll system could pass along the data I need in JSON format via a webhook/API call, alas it can’t do that. I figured that I could write some App Script to parse the emails and grab the data that way. However, before coding, I always like to check if someone has already solved the problem. CloudHQ has an add-on called “Export Email to Sheets” which does exactly what its name suggests. For any email with a given Gmail label (I match on sender and subject) it parses the email and adds the results to a Google Sheet.

As I covered in a previous blog post, I like writing GAM-ADV-XTD3 commands a Google Sheet formulas. I can create an account and send the password to the users personal email with the following Google Sheet formula:

="gam create user " & (left(I2,find("@",I2)-1)) & " firstname """ & A2 & """ lastname """ & B2 & """ password random changepassword on notify " & E2

Here is a key to the fields I’m using:

I2A field with the new school email address we are creating
(left(I2,find("@",I2)-1))Calculates into the part of their email address before the @ symbol
A2A field with the first name of the new hire
B2A field with their last name of the new hire
E2The personal email address of the new hire

To create an account for me:

gam create user ssimon firstname "Shandor" lastname "Simon" password random changepassword on notify shandor@mypersonalemaildomain.com

Once the account is created there are a few other useful GAM commands:

Put the new user in an OU

gam update user ssimon org "/Faculty and Staff"

Add the user to a group

gam update group staff@myschooldomain.org add member user ssimon

GAM, Google Sheets & Calendars

While web interfaces make things easier, it is often much faster to do things on the command line, especially if you are trying to make changes in bulk. 

My school recently added Google Calendars for all our sports teams. One trick to make this easier is to use Google Sheets (or Excel) to convert spreadsheet data into commands I can paste into the terminal window of my Mac. 

I use GAM-ADV-XTD3 (a more advanced version of GAM) that uses the Google Admin APIs to mange my Google Workspace environment from the command line.

In GAM-ADV-XTD3 you can create a calendar in a user account like so:

gam user athletics-calendars create calendar summary "Sailing - Varsity"

In the example athletics-calendars is the username of account I wish to create the account in and Sailing - Varsity is the calendar name.

If I have a Google sheet with a list of sports I need to create a calendars for, I can simply write a formula (like the one below) to generate the commands to create all those calendars.

=" gam user athletics-calendars create calendar summary """ & A1 &""""

This combines the text of the command with the contents of the cell A1 (which is where the sports team’s name is). To quote a quote is a little tricky but the formula above works. I then fill down for all the sports teams and now I have the commands to create all of the calendars. I can highlight all of the cells with these commands (B1:B17 in the example below) and paste them into terminal to run them.

After a few seconds I now have 17 calendars created!

Some other handy e GAM-ADV-XTD3 calendar related commands:

Give a user access/permission to a calendar:

gam calendar xyzzyasfda@group.calendar.google.com add acls writer user@exampledomain.com

xyzzyasfda@group.calendar.google.com is the Calendar ID that you want to give access to user@exampledomain.com. While writer allows someone to edit a calendar, you need to assign owner in order for them to change sharing settings.

Make a calendar public:

gam calendar xyzzyasfda@group.calendar.google.com add acls reader default

If you want to make a calendar public (so that anyone on the can access it) you use default as the username. Using reader provides read-only access.

List all of the calendar in a users account:

gam user sample_user print calendars showhidden todrive

This lists all of the calendars in sample_user's account in a new Google Sheet (todrive). This can be handy to find the Calendar ID of a specific calendar.

Meraki API

So Cisco/Meraki has some pretty good documentation at using the Meraki API with Python. Short version is use pip to install Meraki (pip install meraki) and then in your shell set an environmental variable called MERAKI_DASHBOARD_API_KEY to your Meraki API key (export MERAKI_DASHBOARD_API_KEY=xxxxxxx) which can be enabled and found (see https://documentation.meraki.com/zGeneral_Administration/Other_Topics/The_Cisco_Meraki_Dashboard_API). Below is a little script to grab information about the APs in your network. You’ll need to specify the network name in the Meraki Dashboard. The script assumes there is only one organization, if you have more than one, you’ll have to loop through those to find the id of your organization. The script also makes calls to getDeviceWirelessRadioSettings and getDeviceWirelessStatus which provides more information about those APs, but I don’t make any efforts to format or do anything with the results.

import meraki

dashboard = meraki.DashboardAPI()
#name of my network that has APs
ap_network_name = "xyxy"

# Get orgs linked to API key
my_orgs = dashboard.organizations.getOrganizations()
# Get networks linked to org. Assuming there is only one org (true in my case)
my_networks = dashboard.organizations.getOrganizationNetworks(organizationId=my_orgs[0]["id"])
# Find the network ID of the network with my network name
for netw in my_networks:
	if (netw["name"] == ap_network_name):
		my_netw = netw["id"]

my_devices = dashboard.networks.getNetworkDevices(networkId=my_netw)

for a_device in my_devices:
	try:
		# Filter out the devices that aren't APs
		if(a_device["firmware"][:8] == "wireless"):
			# print some basic info on the APs
			print(a_device["name"],a_device["serial"],a_device["firmware"],a_device["model"],a_device["firmware"])
			serial = a_device["serial"]
			# get some info on the radio settings for the AP
			wireless = dashboard.wireless.getDeviceWirelessRadioSettings(serial)
			print(wireless)
			# get some more detailed info on the AP
			wifi_status = dashboard.wireless.getDeviceWirelessStatus(serial)
			print(wifi_status)
	except:
		print("error for", a_device["serial"])
# print(my_devices)

Using ThankView API to download videos

At the end of last year, my school used ThankView to record surprise videos for each graduating senior. We had 2-4 teachers record a short video reflecting on their relationship with the student and wishing them well. As I prepare to leave that job, I wanted to make sure the school had an archive of those videos outside of the ThankView website. I couldn’t figure out a good way to download all of the videos within the product (aside from one at a time). Lucikly, their API allows you to bulk download videos.

Here is the code I used to download all the videos into a directory called thankview:

import http.client
import urllib.request
import mimetypes
import json

conn = http.client.HTTPSConnection("api.thankview.com")
payload = ''
headers = {
  'Content-Type': 'application/json',
  'Authorization': 'Bearer *********'
}

for page in range(6):
  uri = "/api/video?limit=100&page=" + str(page)
  conn.request("GET", uri, payload, headers)
  res = conn.getresponse()
  data = res.read()
  conn.close()
  jsonResponse = json.loads(data)
  videos = jsonResponse["data"]["videos"]
  
  for video in videos:
    print("id: ", video["id"])
    print("title: ", video["title"])
    print("video_path: ", video["video_path"])
    filename = "thankview/" + str(video["id"]) + "-" + video["title"] + ".mp4"
    print(filename)
    try:
      urllib.request.urlretrieve(video["video_path"],filename)
    except:
      print("error")

Using the Zoom API to deal with webinar panelists

In response to the pandemic my school is using Zoom Webinars for various performances. Our student Improv group has done a number of performances. The cast all need to be “panelists” in Zoom parlance, and when you duplicate a Webinar it does not duplicate the panelists. That’s a pain since there are a lot of panelists, and for each you need a name and e-mail address.

I finally got tired of dealing with it and spent a few minutes figuring out the Zoom API. If I ever have a little more time, I’ll make a Google Apps Manager (GAM) knock off that I’ll call ZAM! In the meantime, perhaps someone else will find this helpful.

First, Zoom has good documentation of its API. To start, you will need to build an app to get credentials. For my scripts, I’m using JSON Web Tokens (JWT) for authentication. I found this post in the Zoom Dev Forums helpful as a starting point.

Get information on a Zoom user

Here is a short Python script to get information on a user to give a little idea on how to use the API.

import jwt   # pip3 install pyjwt --user
import http.client
import datetime
import json
import sys
 
api_key = '********' # replace with credentials from
api_sec = '********' # your app in Zoom Marketplace
 
# generate JWT
payload = {
'iss': api_key,
'exp': datetime.datetime.utcnow() + datetime.timedelta(hours=2)
}
 
jwt_encoded = str(jwt.encode(payload, api_sec), 'utf-8')
 
 
# call API: get user list
conn = http.client.HTTPSConnection("api.zoom.us")
headers = {
'authorization': "Bearer %s" % jwt_encoded,
'content-type': "application/json"
}
email = str(sys.argv[1])
conn.request("GET", "/v2/users/" +  email, headers=headers)
res = conn.getresponse()
response_string = res.read().decode('utf-8')
 
user_json = json.loads(response_string)
json_formatted_str = json.dumps(user_json, indent=2)
print(json_formatted_str)

You can use this like this:

# python3.7 zoom-user.py user@domain.com

{
  "id": "xxxxxxxxxxx",
  "first_name": "John",
  "last_name": "Smith",
  "email": "user@domain.com",
  "type": 2,
  "role_name": "Admin",
  "pmi": 111111111,
  "use_pmi": false,
  "personal_meeting_url": "https://myschool.zoom.us/j/111111111?pwd=UmVTASDFc3pXUTNpOFZNb34A09",
  "timezone": "America/Chicago",
  "verified": 0,
  "dept": "",
  "created_at": "2020-04-13T02:42:31Z",
  "last_login_time": "2020-05-22T19:38:06Z",
  "last_client_version": "4.6.20561.0413(mac)",
  "pic_url": "https://lh3.googleusercontent.com/a-/AOh14GgQadfagrA8l4U9fhsbZ1hOTMSvDxE3gGQ",
  "host_key": "111861",
  "jid": "3timqwabrvs00ayurhg_ha@xmpp.zoom.us",
  "group_ids": [
    "qCpDwoo5T1qS4cn8RSy8rQ"
  ],
  "im_group_ids": [],
  "account_id": "fhsbZ1cOTMSvDxE3g",
  "language": "en-US",
  "phone_country": "",
  "phone_number": "",
  "status": "active",
  "job_title": "",
  "location": ""
}

Note: I’ve redacted/change all of the ID and info above for obvious reasons.

Getting a list of all webinars panelists

This Python script lists all of the panelist names and email addresses.

import jwt   # pip3 install pyjwt --user
import http.client
import datetime
import json
import sys

api_key = '********' # replace with credentials from
api_sec = '********' # your app in Zoom Marketplace

# generate JWT
payload = {
'iss': api_key,
'exp': datetime.datetime.utcnow() + datetime.timedelta(hours=2)
}

jwt_encoded = str(jwt.encode(payload, api_sec), 'utf-8')

# call API: get webinar panelists
conn = http.client.HTTPSConnection("api.zoom.us")
headers = {
'authorization': "Bearer %s" % jwt_encoded,
'content-type': "application/json"
}
webinarid = str(sys.argv[1])
conn.request("GET", "/v2/webinars/" + webinarid + "/panelists", headers=headers)
res = conn.getresponse()
response_string = res.read().decode('utf-8')
response_json = json.loads(response_string)

for panelist in response_json["panelists"]:
    print(panelist["name"],  panelist["email"])

Here is an example of the script in action listing panelists for webinar id 91119118113:

# python3.7 zoom-webinar-panelists.py 91119118113
John Smith jsmith@domain.com
Jane Smith jane.smith@anotherdomain.com

Adding Panelists to a Webinar

Next is a Python script to add a panelist to a webinar. It takes the webinar ID, panelist name and email address as parameters.

import jwt   # pip3 install pyjwt --user
import http.client
import datetime
import json
import sys

api_key = '********' # replace with credentials from
api_sec = '********' # your app in Zoom Marketplace

# generate JWT
payload = {
'iss': api_key,
'exp': datetime.datetime.utcnow() + datetime.timedelta(hours=2)
}

jwt_encoded = str(jwt.encode(payload, api_sec), 'utf-8')

# call API: add panelist
conn = http.client.HTTPSConnection("api.zoom.us")
headers = {
'authorization': "Bearer %s" % jwt_encoded,
'content-type': "application/json"
}

num_arguments = len(sys.argv)-1
webinar = str(sys.argv[1])
email = sys.argv[num_arguments]
name = ""
i = 2
while i < num_arguments:
    name = name + sys.argv[i] + " "
    i = i+1
name = name.rstrip()
print(name, email,webinar)
body = "{\"panelists\":[{\"name\":\"" + name + "\",\"email\":\"" + email + "\"}]}"

webinar = str(sys.argv[1])
conn.request("POST", "/v2/webinars/" + webinar + "/panelists", body, headers=headers)
res = conn.getresponse()
response_string = res.read().decode('utf-8')

Here is an exampling of using the script to add “John Q. Public” john.q.public@domain.com to webinar id 91119118113:

python3.7 zoom-webinar-panelists-add.py 91119118113 John Q. Public john.q.public@domain.com

{"id":"91119118113","updated_at":"2020-05-25T23:25:22Z"}

Closing Thoughts

Using these ideas in these two scripts, I can easily copy panelists from one webinar to another. Hopefully this is enough to get you started using the Zoom API. As I mentioned at the start, I think it would be awesome to have a tool like GAM but for Zoom. I’d much rather manage Zoom from the command line.

Cropping PDFs with AppleScript

Due to Coronavirus the printing of our yearbooks was delayed, so we decided to distribute it digitally until the print version was available. There are lots of tools, like ISSUU, Uberflip, or even Google Drive that can do a decent job of displaying a PDF, password protecting it, and preventing it from being downloaded. The challenge was to get the galley proofs with crop marks, etc. into the right format. Further complicating things were that the odd/even pages had crop markets that were offset, so they had to be cropped differently.

Here is an example of a page with the dotted crop marks and the metadata on the bottom.

Fortunately Adobe Acrobat is scriptable. Normally, I tend to avoid installing Acrobat, in favor of using Preview, but its support for scripting is surprisingly good. The follow AppleScript crops odd/even pages differently and solved my problem. (based on https://macscripter.net/viewtopic.php?id=32586)

tell application "Adobe Acrobat"
activate tell active doc repeat with i from 1 to count of pages tell page i set {L, T, R, B} to media box if ((i mod 2) = 0) then set crop box to {L + 37, T - 39, R, B + 156} else set crop box to {L, T - 39, R - 37, B + 156} end if end tell end repeat end tell end tell

Virtual Graduations & End of Year Events in the time or Coronavirus: Stream Deck

In a previous post, I mentioned that I used Wirecast to do live video production of events I broadcast using Zoom. Normally, I’d like to have someone helping me, but that’s much harder when everyone is remote. Fortunately, I found the Elgato StreamDeck to help. It’s a USB hardware device with a bunch of buttons. Each button can trigger an action. Each button acts as a little screen that can be customized. There is a rich ecosystem of plugins for Strem Deck that make it work with other tools.

Here is what it looks like from a recent production:

Most of the blue buttons are clips in Wirecast and the bottom left button is the “play” button for Wirecast. The little red dot on the top left clip shows that it is live. A green dot on another clip shows that it is “next.” I really like the Wirecast plugin for Stream Deck since it gives feedback on what is playing (the red/green dots).

On the bottom right are five buttons that control Zoom (focus/ turn on/off webcam, turn on/off mic, show participants window, show chat window). They are based on the work of LostDomain and make use of Keyboard Maestro, and it’s plug-in for StreamDeck.

The black buttons are for when we went live and I muted the virtual webcam so that the active speaker became the live speaker.

What is nice is that I can control Wirecast while it is in the background and I’m managing Zoom.

Here is another picture of the StreamDeck from another production:

For this production I didn’t use the virtual webcam and instead played videos in QuickTime. The video quality seemed to be a bit better when sharing my screen rather than using a virtual webcam, and video quality was really important for this production.

The two columns of buttons on the right are mostly to control Zoom (except the bottom two). I added a shortcut to the “Share” button and “Record” (again, making use of Keyboard Maestro). The bottom right is a clock that shows the time (including seconds).

All of the other buttons are AppleScripts to control QuickTime Player and Preview. One of the challenges of using QuickTime to play videos in front of an audience (Zoom or in-person) is that they can see you futzing around opening the video and going full-screen. Using AppleScript can really minimize this, especially when triggering it from the StreamDeck.

Here is a little AppleScript to open and play fullscreen a Quicktime video (based on https://gist.github.com/biojazzard/2829190):

set unixpath to "/Users/ssimon/Desktop/PA Night Long.mp4"
set macfile to (POSIX file unixpath)
tell application "QuickTime Player"
	activate
	delay 0.5
	open file macfile
	set looping of document 1 to false
	--FullScreen
	--FullScreen
	--set presenting of document 1 to true
	--GetBounds
	present document 1
	play document 1
	
end tell

I also wanted to be able to pause the video:

tell application "QuickTime Player"
	activate
	delay 0.5
	pause document 1
end tell

And to play:

tell application "QuickTime Player"
	activate
	delay 0.5
	present document 1
	play document 1
end tell

I also wanted to close a video (QuickTime Player will open additional videos in tabs when in fullscreen):

tell application "QuickTime Player"
	activate
	close document 4
end tell

You can also open a document in Preview in fullscreen:

set unixpath to "/Users/ssimon/Desktop/roman.jpg"
set macfile to (POSIX file unixpath)

tell application "Preview"
	activate
	open file macfile
	tell application "System Events"
		keystroke "f" using {control down, command down}
	end tell
end tell

The end result is that nobody sees me moving my mouse around to control QuickTime Player or Zoom and I can rapidly play a video.

For the show, I launched the video (via a StreamDeck button / AppleScript), immediately paused it (via a StreamDeck button / AppleScript), used the Stream Deck “Share” button for Zoom to open the screen sharing window, picked the QuickTime Player window, checked “Optimize for full-screen video clip” and “Share computer sound”, and then the “Share Screen” button. Then I hit “play” (via a StreamDeck button / AppleScript). When the video was over, I stopped sharing my screen and let the speaker talk, and when he was done, I repeated the process all over for the next video.

In the future, I’ll probably change the AppleScript not to play the video and leave it paused, to simplify the process.

I haven’t had a chance to play with it, but there is a mobile version of the StreamDeck software for Android and iOS which looks like it could be a cheaper alternative to buying a StreamDeck.

 

Virtual Graduations & End of Year Events in the time or Coronavirus: Zoom

I’m the tech guy for a private K-12 school and have had to do a lot of work to pull off the various end of year events. Many other people have shared content that I’ve found helpful, and I hope in posting this someone else might find it useful. Mostly, I write these blogs for me, to document stuff and help get my thoughts together. I’m going to do a series of posts on the tools and techniques I’ve used. I’ll start with Zoom.

Despite privacy and security concerns, we have been using Zoom for many of our meetings and classes. Many of those concerns are addressed by paying for a Zoom for Education account which has a better privacy policy and security features.

Our licensed Zoom accounts support 300 attendees, but that isn’t enough for end-of-year events. I have upgraded my account to support 1,000 users and we also have a 500 user Webinar account. Zoom can also stream to an outside service like YouTube Live. We have invited students to join an event via Zoom, and parents, and other adults to use the YouTube Live stream.

Webinar vs. Meeting

The main differences between the more expensive Webinar version of Zoom and the regular version:

  1. Meeting attendees cannot share video or audio. You can enable users to unmute themselves, but by default attendees are passive participants.
  2. Webinars have a new role of “Panelist” who can share their audio, video (with permission), or screen. They can also use the chat feature to chat with all panelists, or all panelists and attendees.
  3. Webinar supports a Q&A function that enables attendees to ask questions and for panelists to answers those questions either publicly or privately.
  4. Chat in webinar has some different options, including allowing attendees to only post to attendees or everyone. Direct chat between individual participants is not allowed.
  5. Webinar allows you to force what view attendees see (host view, active speaker view, gallery/grid view).

We use the webinar version for our various performing arts shows. By setting “Set video layout for attendees” to “Follow host view mode” and setting “Hide Non-Video Participants” in the host’s Zoom application settings. This allows us to rapidly hide/show webcams of people we want on stage. Our improv groups used this to great effect, constantly changing who was on stage throughout the show.

For end of year events, if you want everyone to see each other then you should use the standard version of Zoom. If you want to control what attendees see, you should use Webinar.

Virtual Webcams

One very useful trick is to use virtual webcam software. This basically adds a fake webcam to your computer that you can use in Zoom. You can use the virtual webcam to play recorded video or output from live video production software. I’ve used this to mix recorded video into a Zoom without resorting to screen sharing (which has drawbacks, including really slow transitions when starting/stopping sharing, and that it takes over the entire screen).

I primarily use Telestream Wirecast for both live video production and to act as a virtual webcam. They have a tutorial on “Using Virtual Camera Out in a Zoom Meeting” which gives a pretty good overview of the process. I’ve also experimented with using ManyCam.

I’ve used Wirecast and a virtual webcam to have slides and video side-by-side in some end of year events. This allows me to make the speaker (in the video) the same size or larger than the slides.

WARNING: security features in macOS Catalina and newer versions of Zoom broke virtual webcam support. Apparently, there is a new version of Zoom that is supposed to come out today (5/22) that fixes this. Prior to that, your only choices were to downgrade to Zoom 4.6.x or to run the following from the command line:

codesign --remove-signature /Applications/zoom.us.app/

Streaming to YouTube Live

Sometimes it is really helpful to stream to more people than your Zoom account can support, provide an option for those who don’t want to use Zoom to attend, or to be able to embed a stream within a webpage. Zoom has documentation on how to “Streaming a Meeting or Webinar on YouTube Live.”

Note: you will have to enable your YouTube account for live streaming and it can take 24 hours for the change to go into effect.

For an event, you’ll probably want to know the URL of the stream ahead of time. For that you’ll have to enable Custom Live Streaming instead of using the built-in support for YouTube Live within Zoom.

WARNING: be careful in testing your setup between Zoom and YouTube Live. If you start streaming and then stop, you won’t be able to use that URL again. I now send out links to a landing page on our website instead of the direct link to YouTube. That way I can change the link at the last second if I need to.

 

Using Microsoft PowerBI with the Blackbaud API

At the recent Blackbaud K-12 Conference, I spent some time with Grahm Getty who did a presentation on using PowerBI with the Blackbaud ON products.  The ability to visualize your data is really powerful and can be used to find insights and problems in your data.  Grahm and I spent some time figuring out how to use the Blackbaud API to get the data in real-time into PowerBI so that it is always up-to-date and doesn’t require exporting and using CSV files.  We were able to get this to work with the desktop version of PowerBI, but it doesn’t seem to work with the web version.  I haven’t given up on that, but for now I thought I would share a tutorial on how to get it to work with the desktop version of PowerBI.  What follows is a tutorial that assume you know nothing of the Blackbaud API nor Microsoft PowerBI.

Prerequisites
This tutorial requires:

  1. You have the Blackbaud API enabled.  There is an annual fee for this, so you need to contact your account manager if you do not already have it.  It comes bundled with ODBC access, so if you have that, then you have API access.
  2. A machine running some modern version of Windows.  It can be emulated in VMware or Parallels if your a Mac user.  PowerBI requires Windows.

Setup within the Blackbaud “ON” products

  1. Create a new user within the “ON” products to use to make API calls.  It is best practice to use a separate account to make API calls.
    1. As a Platform Manager, go to Core>Users/Access>Profile
    2. Click “Add New User”
    3. Fill out the Last Name, First Name, Username and pick and confirm a temporary password.  I used “PowerBI” as the First Name, “API” as the Last Name and “powerbi-api” as the username.
    4. In the screen that follows, edit the user.
    5. Under “System Information” for the user we just created, click “Role Membership”.
    6. Click “Edit”.
    7. Enable “Web Services API Manager” and click on “Save & Exit”.
    8. Sign Out of the ON products and login with your newly created account.
    9. When you login, you will be forced to change your password.  Change it.
  2. Sign Out of the ON products and login to your normal account.
  3. Go to lists (Faculty>My Day>Schedule & Performance>Lists>View All).
  4. At this point, we need to create the list that we want to use in PowerBI.  For the purposes of this demo, we will use one based on a list template.
    1. Click on List Templates.
    2. Under “Template Category” pick “Constituent Information”
    3. Find “Students by Current Grade” and click on “View/Copy”
    4. Under name, give it a name (I used “PowerBI student information”)
    5. Let’s add Ethnicity.
    6. Click on the “Select Objects” tab.
    7. Under “Constituent Information”, click “User Citizenship/Residency”
    8. Click on the “Display Fields” tab.
    9. At the bottom of the screen, click on “Select Fields…”
    10. Expand “User Base” by clicking on the +
    11. Expand “User Citizenship/Residency”
    12. Check the checkbox next to “Ethnicity” and “Citizenship” and then click the “Select” button.
    13. Click the “Preview” button on the bottom right of the screen to preview the list.
    14. Click “Save & Exit” on the top right.
  5. Next we need to get the list ID number so that we can use it in an API call.  This ID number is somewhat hidden.
    1. Control-click on the “Run” link next to the list we just created.
    2. Select “Copy Link” from the pop-up menu that appears.
    3. Go to the text editor of your choice and paste the link.  (note: if you only see “Run” when you paste you’ll need to paste it as unformatted text (on a Mac: command-option-shift-v).
    4. You should see something like: javascript:__pdL(‘52586′,’Advanced%20List:%20API%20-%20list%20example’,%20’1′,%20’~slid=49748~ml=False~sln=API%20-%20list%20example’,%20”,%20’False’,%20’0′,%20”,%20’default.aspx’)
    5. after “slid=” there is a number (in the example 49748).  This is the list ID.  Make a note of your list ID.
    6. Next we have to grant the API account we setup earlier with access to the list we want to use.
      1. Click on the “User Access” link to the right of the link. 
      2. Click on “Add User(s)”.
      3. Search for the API user you created.  (I searched for last name of “API). 
      4. To the left of the API user in the search results, click on the blue “>>” link to move it to “Added Users” and then click on “Save & Exit”.
      5. Leave it with the defaults of “Run” and press “Save & Exit” again.

Microsoft PowerBI Setup

  1. If you do not have the desktop version of Microsoft PowerBI installed on your Windows machine, go to https://powerbi.microsoft.com and click on “Sign up for free” in the top right.  Then click “download free”.  Install it on your machine.
  2. Launch PowerBI Desktop (this takes a while on my machine).
  3. Cancel out of the splash screen. 
  4. Press the “Edit Queries” button in the tool ribbon. 
  5. Press the “New Source” button and then select “Blank Query” and press the “Connect” button.
  6. Right click on the new query that was created and select “Advanced Editor”.
  7. PowerBI makes use of a query language that can be used for advanced queries.  Because of how the Blackbaud ON API handles authentication, we need to use this advanced query language.  Copy the code below into the advanced editor, replacing the code in red with your school’s ON website address, API username/password, and the list ID.  When you edit the text, make sure you don’t use smart quotes.

    let
    Source = Json.Document(Web.Contents(“https://myschool.myschoolapp.com/api/authentication/login?username=apiusername&password=apipassword&format=json”)),T = Source[Token],
    GetList = Function.InvokeAfter(()=>Json.Document(Web.Contents(“https://myschool.myschoolapp.com/api/list/listIDnumber/?t=” & T & “&format=json”)), #duration(0,0,0,5))
    in
    GetList

  8. Click on the “Done” button in the Advanced Editor. 
  9. At this point there are a few setting changes related to the PowerBI security model that I don’t quite understand. Next to “Please specify how to connect” press “Edit Credentials.”
  10. Pick “Anonymous” from the list on the left and press “Connect”. 
  11. Next, you should see an alert “Information is required about data Privacy.”  Press the “Continue button. 
  12. In the pop-up to the right of your website URL pick “Public” then press “Save”.
  13. You should then see results as a column of “Records”.  We will need to expand those. 
  14. Right click on the column header and select “To Table”.
  15. Then an icon will appear on the right side of the column header.  Click on it and you will be able to pick what columns of data you want to view from the record.  Uncheck “Use original column name as prefix” and then click “OK”.
  16. Press the “Close & Apply” button on the left side of the tool ribbon.
  17. Now it’s time for us to build our data visualization!
  18. On the right side of the screen, click on the first icon “Stacked bar chart”.
  19. From the right side of the screen, drag the “GradYear” field to the Axis box, and drag “Gender” to “Value” and then drag it again to “Legend”.
  20. We should now see a chart.  My sample data is pretty boring.
  21. Make sure you’ve clicked on the chart and then click on the paint roller icon.
  22. Click on “Data colors” on the right and pick a light pink for “F” and a light blue for “M”. 
  23. Turn on “Data Labels”, change the font to a larger size and switch the position to “Inside Center”.
  24. Resize the box around the chart to make it bigger.
  25. Now click on the stacked bar chart icon again, and put “GradYear” in the axis and “User ID” in the value field.  Note that it automatically switches “User ID” to “Count of User ID”.
  26. Next click on the “Card” icon (it has the 123 on it).  Drag User ID to it’s value box and click on the triangle next to it to switch it to “Count of User ID (Distinct)”.
  27. Now try clicking on one of the graduation years in the chart of them and notice how the count we just added changed and what it did to the other chart.
  28. Finally save your file.