Strings
Note: Functions taking Tensor
arguments can also take anything accepted by
tf.convert_to_tensor
.
[TOC]
Hashing
String hashing ops take a string input tensor and map each element to an integer.
tf.string_to_hash_bucket_fast(input, num_buckets, name=None)
Converts each string in the input Tensor to its hash mod by a number of buckets.
The hash function is deterministic on the content of the string within the
process and will never change. However, it is not suitable for cryptography.
This function may be used when CPU time is scarce and inputs are trusted or
unimportant. There is a risk of adversaries constructing inputs that all hash
to the same bucket. To prevent this problem, use a strong hash function with
tf.string_to_hash_bucket_strong
.
Args:
input
: ATensor
of typestring
. The strings to assign a hash bucket.num_buckets
: Anint
that is>= 1
. The number of buckets.name
: A name for the operation (optional).
Returns:
A Tensor
of type int64
.
A Tensor of the same shape as the input string_tensor
.
tf.string_to_hash_bucket_strong(input, num_buckets, key, name=None)
Converts each string in the input Tensor to its hash mod by a number of buckets.
The hash function is deterministic on the content of the string within the
process. The hash function is a keyed hash function, where attribute key
defines the key of the hash function. key
is an array of 2 elements.
A strong hash is important when inputs may be malicious, e.g. URLs with
additional components. Adversaries could try to make their inputs hash to the
same bucket for a denial-of-service attack or to skew the results. A strong
hash prevents this by making it dificult, if not infeasible, to compute inputs
that hash to the same bucket. This comes at a cost of roughly 4x higher compute
time than tf.string_to_hash_bucket_fast
.
Args:
input
: ATensor
of typestring
. The strings to assign a hash bucket.num_buckets
: Anint
that is>= 1
. The number of buckets.key
: A list ofints
. The key for the keyed hash function passed as a list of two uint64 elements.name
: A name for the operation (optional).
Returns:
A Tensor
of type int64
.
A Tensor of the same shape as the input string_tensor
.
tf.string_to_hash_bucket(string_tensor, num_buckets, name=None)
Converts each string in the input Tensor to its hash mod by a number of buckets.
The hash function is deterministic on the content of the string within the process.
Note that the hash function may change from time to time.
This functionality will be deprecated and it's recommended to use
tf.string_to_hash_bucket_fast()
or tf.string_to_hash_bucket_strong()
.
Args:
string_tensor
: ATensor
of typestring
.num_buckets
: Anint
that is>= 1
. The number of buckets.name
: A name for the operation (optional).
Returns:
A Tensor
of type int64
.
A Tensor of the same shape as the input string_tensor
.
Joining
String joining ops concatenate elements of input string tensors to produce a new string tensor.
tf.reduce_join(inputs, reduction_indices, keep_dims=None, separator=None, name=None)
Joins a string Tensor across the given dimensions.
Computes the string join across dimensions in the given string Tensor of shape
[d_0, d_1, ..., d_n-1]
. Returns a new Tensor created by joining the input
strings with the given separator (default: empty string). Negative indices are
counted backwards from the end, with -1
being equivalent to n - 1
. Passing
an empty reduction_indices
joins all strings in linear index order and outputs
a scalar string.
For example:
# tensor `a` is [["a", "b"], ["c", "d"]]
tf.reduce_join(a, 0) ==> ["ac", "bd"]
tf.reduce_join(a, 1) ==> ["ab", "cd"]
tf.reduce_join(a, -2) = tf.reduce_join(a, 0) ==> ["ac", "bd"]
tf.reduce_join(a, -1) = tf.reduce_join(a, 1) ==> ["ab", "cd"]
tf.reduce_join(a, 0, keep_dims=True) ==> [["ac", "bd"]]
tf.reduce_join(a, 1, keep_dims=True) ==> [["ab"], ["cd"]]
tf.reduce_join(a, 0, separator=".") ==> ["a.c", "b.d"]
tf.reduce_join(a, [0, 1]) ==> ["acbd"]
tf.reduce_join(a, [1, 0]) ==> ["abcd"]
tf.reduce_join(a, []) ==> ["abcd"]
Args:
inputs
: ATensor
of typestring
. The input to be joined. All reduced indices must have non-zero size.reduction_indices
: ATensor
of typeint32
. The dimensions to reduce over. Dimensions are reduced in the order specified. Omittingreduction_indices
is equivalent to passing[n-1, n-2, ..., 0]
. Negative indices from-n
to-1
are supported.keep_dims
: An optionalbool
. Defaults toFalse
. IfTrue
, retain reduced dimensions with length1
.separator
: An optionalstring
. Defaults to""
. The separator to use when joining.name
: A name for the operation (optional).
Returns:
A Tensor
of type string
.
Has shape equal to that of the input with reduced dimensions removed or
set to 1
depending on keep_dims
.
tf.string_join(inputs, separator=None, name=None)
Joins the strings in the given list of string tensors into one tensor;
with the given separator (default is an empty separator).
Args:
inputs
: A list of at least 1Tensor
objects of typestring
. A list of string tensors. The tensors must all have the same shape, or be scalars. Scalars may be mixed in; these will be broadcast to the shape of non-scalar inputs.separator
: An optionalstring
. Defaults to""
. string, an optional join separator.name
: A name for the operation (optional).
Returns:
A Tensor
of type string
.
Splitting
tf.string_split(source, delimiter=' ')
Split elements of source
based on delimiter
into a SparseTensor
.
Let N be the size of source (typically N will be the batch size). Split each
element of source
based on delimiter
and return a SparseTensor
containing the splitted tokens. Empty tokens are ignored.
If delimiter
is an empty string, each element of the source
is split
into individual 1 character strings.
For example: N = 2, source[0] is 'hello world' and source[1] is 'a b c', then the output will be
st.indices = [0, 0; 0, 1; 1, 0; 1, 1; 1, 2] st.shape = [2, 3] st.values = ['hello', 'world', 'a', 'b', 'c']
Args:
source
:1-D
stringTensor
, the strings to split.delimiter
:0-D
stringTensor
, the delimiter character, the string should be length 0 or 1.
Returns:
A SparseTensor
of rank 2
, the strings split according to the delimiter.
The first column of the indices corresponds to the row in source
and the
second column corresponds to the index of the split component in this row.
Raises:
ValueError
: If delimiter is not a character.
Conversion
tf.as_string(input, precision=None, scientific=None, shortest=None, width=None, fill=None, name=None)
Converts each entry in the given tensor to strings. Supports many numeric
types and boolean.
Args:
input
: ATensor
. Must be one of the following types:int32
,int64
,complex64
,float32
,float64
,bool
,int8
.precision
: An optionalint
. Defaults to-1
. The post-decimal precision to use for floating point numbers. Only used if precision > -1.scientific
: An optionalbool
. Defaults toFalse
. Use scientific notation for floating point numbers.shortest
: An optionalbool
. Defaults toFalse
. Use shortest representation (either scientific or standard) for floating point numbers.width
: An optionalint
. Defaults to-1
. Pad pre-decimal numbers to this width. Applies to both floating point and integer numbers. Only used if width > -1.fill
: An optionalstring
. Defaults to""
. The value to pad if width > -1. If empty, pads with spaces. Another typical value is '0'. String cannot be longer than 1 character.name
: A name for the operation (optional).
Returns:
A Tensor
of type string
.
tf.encode_base64(input, pad=None, name=None)
Encode strings into web-safe base64 format.
Refer to the following article for more information on base64 format: en.wikipedia.org/wiki/Base64. Base64 strings may have padding with '=' at the end so that the encoded has length multiple of 4. See Padding section of the link above.
Web-safe means that the encoder uses - and _ instead of + and /.
Args:
input
: ATensor
of typestring
. Strings to be encoded.pad
: An optionalbool
. Defaults toFalse
. Bool whether padding is applied at the ends.name
: A name for the operation (optional).
Returns:
A Tensor
of type string
. Input strings encoded in base64.
tf.decode_base64(input, name=None)
Decode web-safe base64-encoded strings.
Input may or may not have padding at the end. See EncodeBase64 for padding. Web-safe means that input must use - and _ instead of + and /.
Args:
input
: ATensor
of typestring
. Base64 strings to decode.name
: A name for the operation (optional).
Returns:
A Tensor
of type string
. Decoded strings.